HDDS-5183. For AccessControlException do not perform failover.

[bharatviswa504]

What changes were proposed in this pull request?

For AccessControlException do not perform failover, as even with retry it would yield the same result.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5183

How was this patch tested?

Added tests and also ran a sample test in docker.

bash-4.2$ ozone admin scm roles --service-id=scmservice
Exception encountered while connecting to the server
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
	at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
	at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
	at org.apache.hadoop.ipc.Client.call(Client.java:1452)
	at org.apache.hadoop.ipc.Client.call(Client.java:1405)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
	at com.sun.proxy.$Proxy20.submitRequest(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	at com.sun.proxy.$Proxy20.submitRequest(Unknown Source)
	at org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB.submitRequest(SCMSecurityProtocolClientSideTranslatorPB.java:104)
	at org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB.listCACertificate(SCMSecurityProtocolClientSideTranslatorPB.java:324)
	at org.apache.hadoop.hdds.utils.HAUtils.lambda$buildCAList$2(HAUtils.java:414)
	at org.apache.hadoop.hdds.utils.HAUtils.waitForCACerts(HAUtils.java:439)
	at org.apache.hadoop.hdds.utils.HAUtils.buildCAList(HAUtils.java:413)
	at org.apache.hadoop.hdds.utils.HAUtils.buildCAX509List(HAUtils.java:492)
	at org.apache.hadoop.hdds.scm.cli.ContainerOperationClient.newXCeiverClientManager(ContainerOperationClient.java:102)
	at org.apache.hadoop.hdds.scm.cli.ContainerOperationClient.<init>(ContainerOperationClient.java:76)
	at org.apache.hadoop.hdds.scm.cli.ScmOption.createScmClient(ScmOption.java:62)
	at org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:38)
	at org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:29)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
	at picocli.CommandLine.access$1100(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2152)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2530)
	at picocli.CommandLine.parseWithHandler(CommandLine.java:2465)
	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:96)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.lambda$execute$0(OzoneAdmin.java:76)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:159)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.execute(OzoneAdmin.java:74)
	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:87)
	at org.apache.hadoop.hdds.cli.OzoneAdmin.main(OzoneAdmin.java:68)
Can't create SCM client
bash-4.2$ kinit -kt /etc/security/keytabs/HTTP.keytab HTTP/scm2.org@EXAMPLE.COM
bash-4.2$ ozone admin safemode exit
Access denied for user HTTP/scm2.org@EXAMPLE.COM. Superuser privilege is required.

[bshashikant]

Looks good.