HDDS-2032. Ozone client should retry writes in case of any ratis/stateMachine exceptions. #1420

bshashikant · 2019-09-10T07:59:14Z

No description provided.

hadoop-yetus · 2019-09-10T10:27:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	32	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	67	Maven dependency ordering for branch
+1	mvninstall	716	trunk passed
+1	compile	391	trunk passed
+1	checkstyle	73	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	941	branch has no errors when building and testing our client artifacts.
+1	javadoc	166	trunk passed
0	spotbugs	518	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	750	trunk passed
		_ Patch Compile Tests _
0	mvndep	33	Maven dependency ordering for patch
+1	mvninstall	549	the patch passed
+1	compile	384	the patch passed
+1	javac	384	the patch passed
+1	checkstyle	76	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	731	patch has no errors when building and testing our client artifacts.
+1	javadoc	165	the patch passed
-1	findbugs	233	hadoop-hdds generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
		_ Other Tests _
+1	unit	278	hadoop-hdds in the patch passed.
-1	unit	2567	hadoop-ozone in the patch failed.
+1	asflicense	42	The patch does not generate ASF License warnings.
		8828

Reason	Tests
FindBugs	module:hadoop-hdds
	Load of known null value in org.apache.hadoop.hdds.scm.client.HddsClientUtils.checkForException(Exception) At HddsClientUtils.java:in org.apache.hadoop.hdds.scm.client.HddsClientUtils.checkForException(Exception) At HddsClientUtils.java:[line 327]
	Redundant nullcheck of t which is known to be null in org.apache.hadoop.hdds.scm.client.HddsClientUtils.checkForException(Exception) Redundant null check at HddsClientUtils.java:is known to be null in org.apache.hadoop.hdds.scm.client.HddsClientUtils.checkForException(Exception) Redundant null check at HddsClientUtils.java:[line 327]
Failed junit tests	hadoop.ozone.container.TestContainerReplication
	hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
	hadoop.ozone.client.rpc.TestBlockOutputStream
	hadoop.ozone.scm.TestContainerSmallFile
	hadoop.ozone.TestSecureOzoneCluster
	hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
	hadoop.ozone.om.TestOzoneManagerHA
	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/1/artifact/out/Dockerfile
GITHUB PR	#1420
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux dc4b83c91d1e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `bc2d3a7`
Default Java	1.8.0_222
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/1/artifact/out/new-findbugs-hadoop-hdds.html
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/1/testReport/
Max. process+thread count	4598 (vs. ulimit of 5500)
modules	C: hadoop-hdds/client hadoop-ozone/client U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

mukul1987

The patch generally looks good to me. There are some findbugs issues. can you please address those.

mukul1987 · 2019-09-10T17:34:35Z

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java

-    } else if (retryFailure || t instanceof TimeoutException
-        || t instanceof GroupMismatchException
-        || t instanceof NotReplicatedException) {
+    } else {


So apart from SCE, all exceptions are expected to be related to the pipeline ?

yes...If dn reports an StorageContainerException , its specific to containers in dns but other that if ratis reports any other exceptions , it implies issues in the pipeline itself

hadoop-yetus · 2019-09-16T07:57:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	50	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	77	Maven dependency ordering for branch
-1	mvninstall	40	hadoop-ozone in trunk failed.
-1	compile	25	hadoop-ozone in trunk failed.
+1	checkstyle	79	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	1034	branch has no errors when building and testing our client artifacts.
+1	javadoc	160	trunk passed
0	spotbugs	216	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	24	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	26	Maven dependency ordering for patch
-1	mvninstall	36	hadoop-ozone in the patch failed.
-1	compile	25	hadoop-ozone in the patch failed.
-1	javac	25	hadoop-ozone in the patch failed.
-0	checkstyle	31	hadoop-hdds: The patch generated 2 new + 40 unchanged - 3 fixed = 42 total (was 43)
-0	checkstyle	33	hadoop-ozone: The patch generated 2 new + 144 unchanged - 2 fixed = 146 total (was 146)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	766	patch has no errors when building and testing our client artifacts.
+1	javadoc	90	hadoop-hdds in the patch passed.
+1	javadoc	91	hadoop-ozone generated 0 new + 255 unchanged - 2 fixed = 255 total (was 257)
-1	findbugs	24	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	139	hadoop-hdds in the patch failed.
-1	unit	25	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		3670

Reason	Tests
Failed junit tests	hadoop.ozone.container.ozoneimpl.TestOzoneContainer
	hadoop.ozone.container.keyvalue.TestKeyValueContainer

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/Dockerfile
GITHUB PR	#1420
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 1cf2b6245356 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `85b1c72`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/branch-compile-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/patch-compile-hadoop-ozone.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/diff-checkstyle-hadoop-hdds.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/diff-checkstyle-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/testReport/
Max. process+thread count	336 (vs. ulimit of 5500)
modules	C: hadoop-hdds/client hadoop-ozone/client U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

mukul1987 · 2019-09-17T15:40:47Z

Thanks for working on this @bshashikant , there are some conflicts with this patch. Can you please rebase.

hadoop-yetus · 2019-09-17T19:15:45Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	78	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	31	Maven dependency ordering for branch
-1	mvninstall	28	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-ozone in trunk failed.
+1	checkstyle	51	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	943	branch has no errors when building and testing our client artifacts.
+1	javadoc	159	trunk passed
0	spotbugs	173	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	23	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	23	Maven dependency ordering for patch
-1	mvninstall	30	hadoop-ozone in the patch failed.
-1	compile	21	hadoop-ozone in the patch failed.
-1	javac	21	hadoop-ozone in the patch failed.
-0	checkstyle	25	hadoop-hdds: The patch generated 2 new + 40 unchanged - 3 fixed = 42 total (was 43)
-0	checkstyle	27	hadoop-ozone: The patch generated 2 new + 144 unchanged - 2 fixed = 146 total (was 146)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	728	patch has no errors when building and testing our client artifacts.
+1	javadoc	66	hadoop-hdds in the patch passed.
+1	javadoc	83	hadoop-ozone generated 0 new + 253 unchanged - 2 fixed = 253 total (was 255)
-1	findbugs	23	hadoop-ozone in the patch failed.
		_ Other Tests _
+1	unit	263	hadoop-hdds in the patch passed.
-1	unit	25	hadoop-ozone in the patch failed.
+1	asflicense	30	The patch does not generate ASF License warnings.
		3399

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/Dockerfile
GITHUB PR	#1420
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux ea6521a1b3ea 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / eefe9bc
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/branch-compile-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/patch-compile-hadoop-ozone.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/diff-checkstyle-hadoop-hdds.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/diff-checkstyle-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/testReport/
Max. process+thread count	438 (vs. ulimit of 5500)
modules	C: hadoop-hdds/client hadoop-ozone/client U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1420/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

mukul1987

Thanks for updating the patch @bshashikant. +1 the patch looks good to me.

…eMachine exceptions. Contributed by Shashikant Banerjee (apache#1420).

bshashikant requested a review from mukul1987 September 10, 2019 07:59

mukul1987 added the ozone label Sep 10, 2019

mukul1987 reviewed Sep 10, 2019

View reviewed changes

bshashikant closed this Sep 17, 2019

bshashikant force-pushed the HDDS-2032 branch from 809d373 to c474e24 Compare September 17, 2019 16:20

Rebased to latest trunk

0aab0a8

bshashikant reopened this Sep 17, 2019

mukul1987 approved these changes Sep 18, 2019

View reviewed changes

bshashikant merged commit 2c52d00 into apache:trunk Sep 18, 2019

bshashikant deleted the HDDS-2032 branch September 18, 2019 17:00

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HDDS-2032. Ozone client should retry writes in case of any ratis/stat…

02da4e9

…eMachine exceptions. Contributed by Shashikant Banerjee (apache#1420).

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HDDS-2032. Ozone client should retry writes in case of any ratis/stat…

83f9e43

…eMachine exceptions. Contributed by Shashikant Banerjee (apache#1420).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-2032. Ozone client should retry writes in case of any ratis/stateMachine exceptions. #1420

HDDS-2032. Ozone client should retry writes in case of any ratis/stateMachine exceptions. #1420

bshashikant commented Sep 10, 2019

hadoop-yetus commented Sep 10, 2019

mukul1987 left a comment

mukul1987 Sep 10, 2019

bshashikant Sep 11, 2019

hadoop-yetus commented Sep 16, 2019

mukul1987 commented Sep 17, 2019

hadoop-yetus commented Sep 17, 2019

mukul1987 left a comment

HDDS-2032. Ozone client should retry writes in case of any ratis/stateMachine exceptions. #1420

HDDS-2032. Ozone client should retry writes in case of any ratis/stateMachine exceptions. #1420

Conversation

bshashikant commented Sep 10, 2019

hadoop-yetus commented Sep 10, 2019

mukul1987 left a comment

Choose a reason for hiding this comment

mukul1987 Sep 10, 2019

Choose a reason for hiding this comment

bshashikant Sep 11, 2019

Choose a reason for hiding this comment

hadoop-yetus commented Sep 16, 2019

mukul1987 commented Sep 17, 2019

hadoop-yetus commented Sep 17, 2019

mukul1987 left a comment

Choose a reason for hiding this comment