HDDS-9635. Trying to close a closed container from CLI results in indefinite retries. by sadanand48 · Pull Request #5710 · apache/ozone

sadanand48 · 2023-12-01T07:49:47Z

What changes were proposed in this pull request?

SCM should let client know if container is closed when it tries to close a container instead of retrying.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9635

How was this patch tested?

Unit tests

…efinite retries (cherry picked from commit bfafdfc98191054873f11ff94239bf1a2852e1a2)

adoroszlai

Thanks @sadanand48 for the patch.

hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto

...pache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java

adoroszlai · 2023-12-04T14:07:21Z

Thanks @sadanand48 for updating the patch.

Now that I've tested it, I noticed that while the container is closing, we may still get some ugly exceptions a few times, before finally getting Container ... already closed:

com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.scm.exceptions.SCMException): Cannot close a CLOSING container.
  at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.closeContainer(SCMClientProtocolServer.java:669)
  ...
, while invoking $Proxy20.submitRequest over nodeId=scmNodeId,nodeAddress=scm/172.19.0.8:9860. Trying to failover after sleeping for 2000ms. Current retry count: 0.
...
Container 2 already closed

I also noticed that after Container ... already closed, the command exits with error code.

I would like to suggest considering the following enhancements if possible:

show Container ... already (CLOSING|CLOSED) (only the actual state, not both)
let the command exit successfully

What do you think?

sadanand48 · 2023-12-04T16:36:02Z

Thanks, it makes sense, will update the patch.

xBis7

@sadanand48 Thanks for the patch.

How does the client know when to retry and when not to? Does it have to do with the exception type or a non-0 exit code?

Will it be better to return the response proto back to the client instead of returning void and throwing an exception?

sadanand48 · 2023-12-08T08:53:34Z

Command exits now

bash-4.2$ ozone admin container close 1
Unable to close container
java.io.IOException: Container 1 is in closing state
	at org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB.closeContainer(StorageContainerLocationProtocolClientSideTranslatorPB.java:592)
	at org.apache.hadoop.hdds.scm.cli.ContainerOperationClient.closeContainer(ContainerOperationClient.java:423)
	at org.apache.hadoop.hdds.scm.cli.container.CloseSubcommand.execute(CloseSubcommand.java:53)
	at org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:39)
	at org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:29)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.lambda$execute$0(OzoneAdmin.java:92)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:169)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:149)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.execute(OzoneAdmin.java:91)
	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.main(OzoneAdmin.java:84)
bash-4.2$ ozone admin container close 1
Unable to close container
java.io.IOException: Container 1 already closed
	at org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB.closeContainer(StorageContainerLocationProtocolClientSideTranslatorPB.java:592)
	at org.apache.hadoop.hdds.scm.cli.ContainerOperationClient.closeContainer(ContainerOperationClient.java:423)
	at org.apache.hadoop.hdds.scm.cli.container.CloseSubcommand.execute(CloseSubcommand.java:53)
	at org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:39)
	at org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:29)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.lambda$execute$0(OzoneAdmin.java:92)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:169)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:149)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.execute(OzoneAdmin.java:91)
	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.main(OzoneAdmin.java:84)

sadanand48 · 2023-12-08T08:57:45Z

Will it be better to return the response proto back to the client instead of returning void and throwing an exception?

Thanks @xBis7 , yes the current patch returns the exception state (CLOSED/CLOSING) in the response proto and lets client know to not retry. In regular case, the retry happens whenever there is an exception.

adoroszlai · 2023-12-08T09:14:01Z

Command exits now

bash-4.2$ ozone admin container close 1
Unable to close container
java.io.IOException: Container 1 is in closing state
	at org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB.closeContainer(StorageContainerLocationProtocolClientSideTranslatorPB.java:592)
...

@sadanand48, I think we should print these stack traces only in verbose mode (--verbose).

sadanand48 · 2023-12-11T07:47:16Z

@sadanand48, I think we should print these stack traces only in verbose mode (--verbose).

Thanks @adoroszlai , updated.

bash-4.2$ ozone admin container close 1
Unable to close container : Container 1 already closed
bash-4.2$ ozone admin --verbose container close 1
Unable to close container
java.io.IOException: Container 1 already closed
	at org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB.closeContainer(StorageContainerLocationProtocolClientSideTranslatorPB.java:592)
	at org.apache.hadoop.hdds.scm.cli.ContainerOperationClient.closeContainer(ContainerOperationClient.java:423)
	at org.apache.hadoop.hdds.scm.cli.container.CloseSubcommand.execute(CloseSubcommand.java:58)
	at org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:39)
	at org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:29)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.lambda$execute$0(OzoneAdmin.java:92)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:169)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:149)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.execute(OzoneAdmin.java:91)
	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.main(OzoneAdmin.java:84)

xBis7

@sadanand48 I've tested it locally. LGTM!

adoroszlai · 2023-12-11T18:57:34Z

Thanks @sadanand48 for updating the patch, LGTM.

adoroszlai · 2023-12-11T18:57:58Z

@errose28 @kaijchen @sodonnel would you like to take a look?

adoroszlai · 2023-12-13T08:23:53Z

Thanks @sadanand48 for the patch, @xBis7 for the review.

Sadanand Shenoy added 2 commits December 1, 2023 00:20

HDDS-9635. Trying to close a closed container from CLI results in ind…

bf9b753

…efinite retries (cherry picked from commit bfafdfc98191054873f11ff94239bf1a2852e1a2)

fix test failure

a7f21f6

adoroszlai reviewed Dec 1, 2023

View reviewed changes

hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto Outdated Show resolved Hide resolved

...pache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java Outdated Show resolved Hide resolved

address comments

090bf0e

adoroszlai requested a review from xBis7 December 4, 2023 14:10

xBis7 reviewed Dec 4, 2023

View reviewed changes

Sadanand Shenoy added 2 commits December 8, 2023 13:46

address comments

92539b2

update proto

0ee64ba

address comment

a895195

xBis7 approved these changes Dec 11, 2023

View reviewed changes

adoroszlai merged commit 09ec0b1 into apache:master Dec 13, 2023

Conversation

sadanand48 commented Dec 1, 2023

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

adoroszlai commented Dec 4, 2023

Uh oh!

sadanand48 commented Dec 4, 2023

Uh oh!

xBis7 left a comment

Choose a reason for hiding this comment

Uh oh!

sadanand48 commented Dec 8, 2023

Uh oh!

sadanand48 commented Dec 8, 2023

Uh oh!

adoroszlai commented Dec 8, 2023

Uh oh!

sadanand48 commented Dec 11, 2023

Uh oh!

xBis7 left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Dec 11, 2023

Uh oh!

adoroszlai commented Dec 11, 2023

Uh oh!

adoroszlai commented Dec 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants