java.lang.SecurityException: "putProviderProperty.SaslPlainServer" and "insertProvider.SaslPlainServer" for :Plugin Repository HDFS

[risdenk]

Elasticsearch version (bin/elasticsearch --version):
Version: 5.6.2, Build: 57e20f3/2017-09-23T13:16:45.703Z, JVM: 1.8.0_121
and
Version: 6.0.0-rc1, Build: b9c0df2/2017-09-25T19:11:45.815Z, JVM: 1.8.0_121

Plugins installed:

• repository-hdfs
• x-pack

JVM version (java -version):
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-tdc1-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux HOSTNAME 3.0.101-0.113.TDC.1.R.0-default #1 SMP Fri Dec 9 04:51:20 PST 2016 (ca32437) x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Elasticsearch plugin HDFS repository fails to create repositories, with the following two errors java.security.AccessControlException: access denied ("java.security.SecurityPermission" "putProviderProperty.SaslPlainServer") and java.security.AccessControlException: access denied ("java.security.SecurityPermission" "insertProvider.SaslPlainServer") from the JVM security manager.

I worked around each in turn by adding to a java.policy file and passing to Elasticsearch on startup. The second permission error was only found after adding an exception for the first one.

Steps to reproduce:

1. Install Elasticsearch
2. Install repository-hdfs plugin
3. Create Elasticsearch snapshot repository pointing to HDFS
4. Try to create repository. I am still trying to validate complete reproduction steps

Provide logs (if relevant):
The stacktraces below are from 5.6.2. I can grab from 6.0.0-rc1 if necessary.

Stacktrace from missing security policy permission putProviderProperty.SaslPlainServer

[2017-10-03T11:24:39,212][WARN ][o.e.r.h.HdfsRepository   ] Hadoop authentication method is set to [SIMPLE], but a Kerberos principal is specified. Continuing with [KERBEROS] authentication.
[2017-10-03T11:24:39,246][WARN ][o.e.r.RepositoriesService] [master-HOSTNAME] failed to create repository [hdfs][REPOSITORY]
java.security.AccessControlException: access denied ("java.security.SecurityPermission" "putProviderProperty.SaslPlainServer")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
	at java.security.AccessControlContext.checkPermission2(AccessControlContext.java:538) ~[?:1.8.0_121]
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:481) ~[?:1.8.0_121]
	at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkSecurityAccess(SecurityManager.java:1759) ~[?:1.8.0_121]
	at java.security.Provider.check(Provider.java:658) ~[?:1.8.0_121]
	at java.security.Provider.put(Provider.java:317) ~[?:1.8.0_121]
	at org.apache.hadoop.security.SaslPlainServer$SecurityProvider.<init>(SaslPlainServer.java:41) ~[?:?]
	at org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:181) ~[?:?]
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:581) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:140) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:107) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) ~[?:?]
	at com.sun.proxy.$Proxy34.mkdirs(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2525) ~[?:?]
	at org.apache.hadoop.fs.Hdfs.mkdir(Hdfs.java:311) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:738) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:734) ~[?:?]
	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) ~[?:?]
	at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:741) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:65) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.lambda$execute$0(HdfsBlobStore.java:132) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
	at java.security.AccessController.doPrivileged(AccessController.java:713) ~[?:1.8.0_121]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.execute(HdfsBlobStore.java:129) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.mkdirs(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.<init>(HdfsBlobStore.java:55) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsRepository.doStart(HdfsRepository.java:116) ~[?:?]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.applyClusterState(RepositoriesService.java:303) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.2.jar:5.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2017-10-03T11:24:39,256][WARN ][o.e.r.RepositoriesService] [master-HOSTNAME] failed to create repository [HDFSREPOSITORY]
org.elasticsearch.repositories.RepositoryException: [HDFSREPOSITORY] failed to create repository
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:388) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.applyClusterState(RepositoriesService.java:303) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.2.jar:5.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.security.AccessControlException: access denied ("java.security.SecurityPermission" "putProviderProperty.SaslPlainServer")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
	at java.security.AccessControlContext.checkPermission2(AccessControlContext.java:538) ~[?:1.8.0_121]
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:481) ~[?:1.8.0_121]
	at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkSecurityAccess(SecurityManager.java:1759) ~[?:1.8.0_121]
	at java.security.Provider.check(Provider.java:658) ~[?:1.8.0_121]
	at java.security.Provider.put(Provider.java:317) ~[?:1.8.0_121]
	at org.apache.hadoop.security.SaslPlainServer$SecurityProvider.<init>(SaslPlainServer.java:41) ~[?:?]
	at org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:181) ~[?:?]
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:581) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:140) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:107) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) ~[?:?]
	at com.sun.proxy.$Proxy34.mkdirs(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2525) ~[?:?]
	at org.apache.hadoop.fs.Hdfs.mkdir(Hdfs.java:311) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:738) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:734) ~[?:?]
	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) ~[?:?]
	at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:741) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:65) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.lambda$execute$0(HdfsBlobStore.java:132) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
	at java.security.AccessController.doPrivileged(AccessController.java:713) ~[?:1.8.0_121]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.execute(HdfsBlobStore.java:129) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.mkdirs(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.<init>(HdfsBlobStore.java:55) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsRepository.doStart(HdfsRepository.java:116) ~[?:?]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-5.6.2.jar:5.6.2]
	... 13 more

Stacktrace from missing security policy permission putProviderProperty.SaslPlainServer

[2017-10-03T11:51:58,287][WARN ][o.e.r.h.HdfsRepository   ] Hadoop authentication method is set to [SIMPLE], but a Kerberos principal is specified. Continuing with [KERBEROS] authentication.
[2017-10-03T11:51:58,320][WARN ][o.e.r.RepositoriesService] [master-HOSTNAME] failed to create repository [hdfs][REPOSITORY]
java.security.AccessControlException: access denied ("java.security.SecurityPermission" "insertProvider.SaslPlainServer")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
	at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkSecurityAccess(SecurityManager.java:1759) ~[?:1.8.0_121]
	at java.security.Security.checkInsertProvider(Security.java:862) ~[?:1.8.0_121]
	at java.security.Security.insertProviderAt(Security.java:359) ~[?:1.8.0_121]
	at java.security.Security.addProvider(Security.java:403) ~[?:1.8.0_121]
	at org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:181) ~[?:?]
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:581) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:140) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:107) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) ~[?:?]
	at com.sun.proxy.$Proxy34.mkdirs(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2525) ~[?:?]
	at org.apache.hadoop.fs.Hdfs.mkdir(Hdfs.java:311) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:738) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:734) ~[?:?]
	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) ~[?:?]
	at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:741) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:65) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.lambda$execute$0(HdfsBlobStore.java:132) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
	at java.security.AccessController.doPrivileged(AccessController.java:713) ~[?:1.8.0_121]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.execute(HdfsBlobStore.java:129) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.mkdirs(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.<init>(HdfsBlobStore.java:55) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsRepository.doStart(HdfsRepository.java:116) ~[?:?]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.applyClusterState(RepositoriesService.java:303) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.2.jar:5.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
	Suppressed: java.security.AccessControlException: access denied ("java.security.SecurityPermission" "insertProvider.SaslPlainServer")
		at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
		at java.security.AccessControlContext.checkPermission2(AccessControlContext.java:538) ~[?:1.8.0_121]
		at java.security.AccessControlContext.checkPermission(AccessControlContext.java:481) ~[?:1.8.0_121]
		at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
		at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
		at java.lang.SecurityManager.checkSecurityAccess(SecurityManager.java:1759) ~[?:1.8.0_121]
		at java.security.Security.checkInsertProvider(Security.java:865) ~[?:1.8.0_121]
		at java.security.Security.insertProviderAt(Security.java:359) ~[?:1.8.0_121]
		at java.security.Security.addProvider(Security.java:403) ~[?:1.8.0_121]
		at org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:181) ~[?:?]
		at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:581) ~[?:?]
		at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
		at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
		at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
		at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:140) ~[?:?]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:107) ~[?:?]
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) ~[?:?]
		at com.sun.proxy.$Proxy34.mkdirs(Unknown Source) ~[?:?]
		at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2525) ~[?:?]
		at org.apache.hadoop.fs.Hdfs.mkdir(Hdfs.java:311) ~[?:?]
		at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:738) ~[?:?]
		at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:734) ~[?:?]
		at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) ~[?:?]
		at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:741) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:65) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:62) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore.lambda$execute$0(HdfsBlobStore.java:132) ~[?:?]
		at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
		at java.security.AccessController.doPrivileged(AccessController.java:713) ~[?:1.8.0_121]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore.execute(HdfsBlobStore.java:129) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore.mkdirs(HdfsBlobStore.java:62) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore.<init>(HdfsBlobStore.java:55) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsRepository.doStart(HdfsRepository.java:116) ~[?:?]
		at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.repositories.RepositoriesService.applyClusterState(RepositoriesService.java:303) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.2.jar:5.6.2]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
		at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

[risdenk]

This relates to #26513 in that the work required is probably very similar.

[risdenk]

@jbaiera - FYI on this since you looked at #26513

[risdenk]

My first hunch from looking through the code previously is it relates to:

https://github.com/elastic/elasticsearch/blob/5.6/plugins/repository-hdfs/src/main/java/org/elasticsearch/repositories/hdfs/HdfsSecurityContext.java#L63

It looks like the permissions are added to the policy file (and being properly picked up) but not being passed in the context.

I don't have a test case for this right now but can look it into it later this week.

[jbaiera]

Hi again @risdenk. Thanks for opening this issue.

Since HDFS takes a lot of liberties in regards to what permissions it has during operation (forking processes, reflective calls, access to user principals), we try to limit the permissions it has during regular operations to be a subset of all the permissions in the policy file (which includes all permissions it needs during the client's first set up).

While we do have them included in the policy file by default, the putProviderProperty.SaslPlainServer and insertProvider.SaslPlainServer permissions were considered unneeded for regular client operations after the client was created during testing. Thus these permissions were removed from the list of restricted permissions for client operations. Could you share the HDFS properties used on this issue as well as any information you find while attempting to get a clean reproduction? We'll need a use case that reproduces the errors in order to get a PR up to fix them.

[risdenk]

Yea I'll try to get to reproducing this cleanly, but likely won't be till next week. I will most likely focus on 6.0.0 to reproduce since that environment has no one on it (and also is closest to master to make PRs easier).

My hunch in case this gives any ideas. It looks like the error is with multiple repositories from a quick glance at the logs. The first repository is initialized just fine. The second (and third) are the ones that have the initialization issue. I haven't looked closer as to why this could be. From the last time I looked at the tests, they were all single repository though. Maybe the second repository initialization takes a different path.

[risdenk]

So I have a new hunch from reviewing the stack trace in more detail. I think this has to do with HDFS NameNode high availability. The code being executed is to create a failover proxy between two NameNodes. I don't think this is tested in the test framework since only a single NameNode is used. I am going to try switching the active NameNode in the cluster and see if that makes a difference as well.

The following part of the stacktrace is the same in both cases. It didn't appear in #26513.

...
at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
...

I haven't gotten this to reproduce cleanly yet outside of the environment though.

[jbaiera]

@risdenk That would make sense - After perusing the failover code, I found that when creating an HA-enabled client the HDFS Client code does not perform any operations or create any connections. Instead, the place where all that initialization would normally occur in a non-HA case is chopped off and done lazily when the user code initiates the first operation.

Quick question: Are you using this HA repository as a read-only repository like in #26513 ? I'm wondering if it's read only and the lack of a verification step is causing it to leave the HDFS client uninitialized.

[risdenk]

@jbaiera - Yes all of our HDFS repository for 5.x and 6.x are readonly right now for testing (prevent someone from making a snapshot). We use 2.x to write to the HDFS repositories right now. We will switch from readonly to read/write for 5.x when we finish deployments.

[risdenk]

I was able to reproduce this with a nice portable Docker implementation with NameNode HA. It reproduces with 5.6.2 and 6.0.0-rc1. It doesn't look like this has anything to do with readonly. The failure occurs with a read/write repository. The NameNode HA state is important. The error occurs when the first NameNode contacted is in Standby mode and must failover to the active NameNode. This is lazily done after the repository is setup. I haven't tested this but would be curious to see if the additional permissions are needed later if the NameNode HA state were to switch after initial client setup.

Here is the repo with everything needed to reproduce.

https://github.com/risdenk/elasticsearch_hdfs_kerberos_testing

The master branch has 5.6.2 and the 6.0.0-rc1 branch has 6.0.0-rc1.

The logs from the Travis test run if you don't want to set this up locally is here:

https://travis-ci.org/risdenk/elasticsearch_hdfs_kerberos_testing/branches

I haven't looked into how much effort it would be to make the Vagrant test framework for the HDFS secure tests support NameNode HA. It was a pain to get Docker and Kerberos to play nicely with hostnames.

[risdenk]

Looks like adding the following to MiniHDFS.java might help. Would need to address the fixed ports still. It would be nice to able to specify the ports dynamically in the yml files for the rest-api-spec tests.

MiniDFSNNTopology miniDFSNNTopology = MiniDFSNNTopology.simpleHATopology();
builder.nnTopology(miniDFSNNTopology);

[jbaiera]

Thanks for the footwork toward getting a reproduction. I think you might have hit the nail on the head - If the repository already exists and the leading namenode steps down, allowing a different one to take it's place as leader, the ensuing connection failover would require a whole new client to be initialized underneath the failover manager.

I'm working on getting this all to play nice in the test framework. I don't like the idea of throwing away our current permission rendering in all cases. If a user isn't using secured HDFS, there's no reason to permit any kerberos operations on the client at that time. In the event that a user IS using secured HDFS, we want to make sure that only the logged in user can initiate kerberos connections, and so on.

Most likely what we'll end up doing is sniff the client configuration for HA settings, and if we find any, we'll open up the rendered permissions only in the case of a client failover to allow for the connection to be re-established.

[risdenk]

@jbaiera - Sounds good. Thanks for digging in further. I think I have the minimum required configs necessary to make HDFS HA work (in the docker-compose.yml) and for an Elasticsearch HDFS repository to use HDFS HA in run_test.sh under the variable HDFS_CONFIGS. Let me know if you need anything else to help test this out.