HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem #2261

ayushtkn · 2020-08-29T07:55:33Z

Backport HDFS-15117 along with related Commits
https://issues.apache.org/jira/browse/HDFS-15117

hadoop-yetus · 2020-08-29T11:52:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	16m 0s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+0 🆗	prototool	0m 0s	prototool was not available.
+0 🆗	markdownlint	0m 0s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 8 new or modified test files.
		_ branch-3.2 Compile Tests _
+0 🆗	mvndep	3m 16s	Maven dependency ordering for branch
+1 💚	mvninstall	22m 22s	branch-3.2 passed
+1 💚	compile	1m 49s	branch-3.2 passed
+1 💚	checkstyle	1m 10s	branch-3.2 passed
+1 💚	mvnsite	2m 25s	branch-3.2 passed
+1 💚	shadedclient	15m 58s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 7s	branch-3.2 passed
+0 🆗	spotbugs	1m 0s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 39s	branch-3.2 passed
-0 ⚠️	patch	1m 15s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 18s	the patch passed
+1 💚	compile	1m 46s	the patch passed
+1 💚	cc	1m 46s	the patch passed
+1 💚	javac	1m 46s	the patch passed
-0 ⚠️	checkstyle	1m 7s	hadoop-hdfs-project: The patch generated 5 new + 675 unchanged - 0 fixed = 680 total (was 675)
+1 💚	mvnsite	2m 11s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 10s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 54s	the patch passed
+1 💚	findbugs	5m 52s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 44s	hadoop-hdfs-client in the patch passed.
-1 ❌	unit	114m 41s	hadoop-hdfs in the patch failed.
+1 💚	unit	19m 10s	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 39s	The patch does not generate ASF License warnings.
		235m 35s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestFSImage
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/1/artifact/out/Dockerfile
GITHUB PR	#2261
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc prototool markdownlint
uname	Linux 0db58b457b49 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / `970b9a2`
Default Java	Private Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01
checkstyle	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/1/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/1/testReport/
Max. process+thread count	3847 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…s. Contributed by Kitti Nanasi. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit dd5e7c6)

…re setting, enabling or adding it. Contributed by Kitti Nanasi. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 951cdd7)

…ibuted by Kitti Nanasi. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 5f15a60)

…e coding policy as a parameter. Contributed by Kitti Nanasi. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 14282e3)

hadoop-yetus · 2020-08-30T08:21:05Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	10m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+0 🆗	prototool	0m 1s	prototool was not available.
+0 🆗	markdownlint	0m 1s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 8 new or modified test files.
		_ branch-3.2 Compile Tests _
+0 🆗	mvndep	3m 16s	Maven dependency ordering for branch
+1 💚	mvninstall	22m 23s	branch-3.2 passed
+1 💚	compile	1m 54s	branch-3.2 passed
+1 💚	checkstyle	1m 10s	branch-3.2 passed
+1 💚	mvnsite	2m 27s	branch-3.2 passed
+1 💚	shadedclient	16m 7s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 10s	branch-3.2 passed
+0 🆗	spotbugs	1m 0s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 40s	branch-3.2 passed
-0 ⚠️	patch	1m 15s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 20s	the patch passed
+1 💚	compile	1m 46s	the patch passed
+1 💚	cc	1m 46s	the patch passed
+1 💚	javac	1m 46s	the patch passed
-0 ⚠️	checkstyle	1m 5s	hadoop-hdfs-project: The patch generated 4 new + 675 unchanged - 0 fixed = 679 total (was 675)
+1 💚	mvnsite	2m 10s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 11s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 58s	the patch passed
+1 💚	findbugs	5m 52s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 45s	hadoop-hdfs-client in the patch passed.
-1 ❌	unit	88m 39s	hadoop-hdfs in the patch failed.
-1 ❌	unit	17m 58s	hadoop-hdfs-rbf in the patch failed.
+1 💚	asflicense	0m 40s	The patch does not generate ASF License warnings.
		203m 18s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/2/artifact/out/Dockerfile
GITHUB PR	#2261
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc prototool markdownlint
uname	Linux 5e02ee2787f5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / `970b9a2`
Default Java	Private Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01
checkstyle	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/2/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/2/testReport/
Max. process+thread count	3887 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2020-08-30T08:38:58Z

Test failures aren't related. Checkstyle warnings due to line length, can't be fixed and exist in the original patches as well

ayushtkn

Have marked the changes made, Rest stays same.

ayushtkn · 2020-09-05T07:43:06Z

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestECAdmin.java

Changed this :
Was previously -

public Timeout globalTimeout = new Timeout(300000, TimeUnit.MILLISECONDS);

ayushtkn · 2020-09-05T07:44:20Z

...ain/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java

Changed this -
Was previously

ProtocolStringList policies = req.getPoliciesList();

Due to proto compatibility issue

ayushtkn · 2020-09-05T08:02:54Z

Thsnx @vinayakumarb for the review and the pro-tip for merging this through CLI. This being open since a week, got me anxious. Thanx for sparing time on weekend :-)
As discussed offline, I will rebase the fork and then merge the branch, with last commit having the PR Id by EOD

umamaheswararao · 2020-09-05T15:59:15Z

Oops I missed your review request @ayushtkn . Thanks @vinayakumarb for review.

…System. Contributed by Ayush Saxena. (apache#2261)

hadoop-yetus · 2020-09-06T02:44:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	15m 31s	Docker mode activated.
-1 ❌	patch	0m 9s	#2261 does not apply to branch-3.2. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/3/artifact/out/Dockerfile
GITHUB PR	#2261
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2261/3/console
versions	git=2.7.4
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn requested a review from umamaheswararao August 29, 2020 12:02

kittinanasi added 4 commits August 30, 2020 10:21

HDFS-12946. Add a tool to check rack configuration against EC policie…

a78430a

…s. Contributed by Kitti Nanasi. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit dd5e7c6)

HDFS-14061. Check if the cluster topology supports the EC policy befo…

0dbcb81

…re setting, enabling or adding it. Contributed by Kitti Nanasi. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 951cdd7)

HDFS-14125. Use parameterized log format in ECTopologyVerifier. Contr…

3ba40d0

…ibuted by Kitti Nanasi. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 5f15a60)

HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasur…

9b8eaa0

…e coding policy as a parameter. Contributed by Kitti Nanasi. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 14282e3)

ayushtkn force-pushed the HDFS-15117 branch from 221cc86 to 128491e Compare August 30, 2020 04:56

ayushtkn requested a review from vinayakumarb September 2, 2020 03:09

ayushtkn commented Sep 5, 2020

View reviewed changes

vinayakumarb approved these changes Sep 5, 2020

View reviewed changes

HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFile…

544d6da

…System. Contributed by Ayush Saxena. (apache#2261)

ayushtkn force-pushed the HDFS-15117 branch from 128491e to 544d6da Compare September 6, 2020 02:26

asfgit merged commit 544d6da into apache:branch-3.2 Sep 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem #2261

HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem #2261

Uh oh!

ayushtkn commented Aug 29, 2020

Uh oh!

hadoop-yetus commented Aug 29, 2020

Uh oh!

hadoop-yetus commented Aug 30, 2020

Uh oh!

ayushtkn commented Aug 30, 2020

Uh oh!

ayushtkn left a comment

Uh oh!

ayushtkn Sep 5, 2020

Uh oh!

ayushtkn Sep 5, 2020

Uh oh!

ayushtkn commented Sep 5, 2020

Uh oh!

umamaheswararao commented Sep 5, 2020

Uh oh!

hadoop-yetus commented Sep 6, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem #2261

HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem #2261

Uh oh!

Conversation

ayushtkn commented Aug 29, 2020

Uh oh!

hadoop-yetus commented Aug 29, 2020

Uh oh!

hadoop-yetus commented Aug 30, 2020

Uh oh!

ayushtkn commented Aug 30, 2020

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

ayushtkn Sep 5, 2020

Choose a reason for hiding this comment

Uh oh!

ayushtkn Sep 5, 2020

Choose a reason for hiding this comment

Uh oh!

ayushtkn commented Sep 5, 2020

Uh oh!

umamaheswararao commented Sep 5, 2020

Uh oh!

hadoop-yetus commented Sep 6, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants