New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

java.lang.SecurityException: "putProviderProperty.SaslPlainServer" and "insertProvider.SaslPlainServer" for :Plugin Repository HDFS #26868

Closed
risdenk opened this Issue Oct 3, 2017 · 20 comments

Comments

Projects
None yet
3 participants
@risdenk
Contributor

risdenk commented Oct 3, 2017

Elasticsearch version (bin/elasticsearch --version):
Version: 5.6.2, Build: 57e20f3/2017-09-23T13:16:45.703Z, JVM: 1.8.0_121
and
Version: 6.0.0-rc1, Build: b9c0df2/2017-09-25T19:11:45.815Z, JVM: 1.8.0_121

Plugins installed:

  • repository-hdfs
  • x-pack

JVM version (java -version):
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-tdc1-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux HOSTNAME 3.0.101-0.113.TDC.1.R.0-default #1 SMP Fri Dec 9 04:51:20 PST 2016 (ca32437) x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Elasticsearch plugin HDFS repository fails to create repositories, with the following two errors java.security.AccessControlException: access denied ("java.security.SecurityPermission" "putProviderProperty.SaslPlainServer") and java.security.AccessControlException: access denied ("java.security.SecurityPermission" "insertProvider.SaslPlainServer") from the JVM security manager.

I worked around each in turn by adding to a java.policy file and passing to Elasticsearch on startup. The second permission error was only found after adding an exception for the first one.

Steps to reproduce:

  1. Install Elasticsearch
  2. Install repository-hdfs plugin
  3. Create Elasticsearch snapshot repository pointing to HDFS
  4. Try to create repository. I am still trying to validate complete reproduction steps

Provide logs (if relevant):
The stacktraces below are from 5.6.2. I can grab from 6.0.0-rc1 if necessary.

Stacktrace from missing security policy permission putProviderProperty.SaslPlainServer

[2017-10-03T11:24:39,212][WARN ][o.e.r.h.HdfsRepository   ] Hadoop authentication method is set to [SIMPLE], but a Kerberos principal is specified. Continuing with [KERBEROS] authentication.
[2017-10-03T11:24:39,246][WARN ][o.e.r.RepositoriesService] [master-HOSTNAME] failed to create repository [hdfs][REPOSITORY]
java.security.AccessControlException: access denied ("java.security.SecurityPermission" "putProviderProperty.SaslPlainServer")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
	at java.security.AccessControlContext.checkPermission2(AccessControlContext.java:538) ~[?:1.8.0_121]
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:481) ~[?:1.8.0_121]
	at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkSecurityAccess(SecurityManager.java:1759) ~[?:1.8.0_121]
	at java.security.Provider.check(Provider.java:658) ~[?:1.8.0_121]
	at java.security.Provider.put(Provider.java:317) ~[?:1.8.0_121]
	at org.apache.hadoop.security.SaslPlainServer$SecurityProvider.<init>(SaslPlainServer.java:41) ~[?:?]
	at org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:181) ~[?:?]
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:581) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:140) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:107) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) ~[?:?]
	at com.sun.proxy.$Proxy34.mkdirs(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2525) ~[?:?]
	at org.apache.hadoop.fs.Hdfs.mkdir(Hdfs.java:311) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:738) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:734) ~[?:?]
	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) ~[?:?]
	at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:741) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:65) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.lambda$execute$0(HdfsBlobStore.java:132) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
	at java.security.AccessController.doPrivileged(AccessController.java:713) ~[?:1.8.0_121]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.execute(HdfsBlobStore.java:129) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.mkdirs(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.<init>(HdfsBlobStore.java:55) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsRepository.doStart(HdfsRepository.java:116) ~[?:?]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.applyClusterState(RepositoriesService.java:303) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.2.jar:5.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2017-10-03T11:24:39,256][WARN ][o.e.r.RepositoriesService] [master-HOSTNAME] failed to create repository [HDFSREPOSITORY]
org.elasticsearch.repositories.RepositoryException: [HDFSREPOSITORY] failed to create repository
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:388) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.applyClusterState(RepositoriesService.java:303) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.2.jar:5.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.security.AccessControlException: access denied ("java.security.SecurityPermission" "putProviderProperty.SaslPlainServer")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
	at java.security.AccessControlContext.checkPermission2(AccessControlContext.java:538) ~[?:1.8.0_121]
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:481) ~[?:1.8.0_121]
	at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkSecurityAccess(SecurityManager.java:1759) ~[?:1.8.0_121]
	at java.security.Provider.check(Provider.java:658) ~[?:1.8.0_121]
	at java.security.Provider.put(Provider.java:317) ~[?:1.8.0_121]
	at org.apache.hadoop.security.SaslPlainServer$SecurityProvider.<init>(SaslPlainServer.java:41) ~[?:?]
	at org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:181) ~[?:?]
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:581) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:140) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:107) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) ~[?:?]
	at com.sun.proxy.$Proxy34.mkdirs(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2525) ~[?:?]
	at org.apache.hadoop.fs.Hdfs.mkdir(Hdfs.java:311) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:738) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:734) ~[?:?]
	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) ~[?:?]
	at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:741) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:65) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.lambda$execute$0(HdfsBlobStore.java:132) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
	at java.security.AccessController.doPrivileged(AccessController.java:713) ~[?:1.8.0_121]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.execute(HdfsBlobStore.java:129) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.mkdirs(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.<init>(HdfsBlobStore.java:55) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsRepository.doStart(HdfsRepository.java:116) ~[?:?]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-5.6.2.jar:5.6.2]
	... 13 more

Stacktrace from missing security policy permission putProviderProperty.SaslPlainServer

[2017-10-03T11:51:58,287][WARN ][o.e.r.h.HdfsRepository   ] Hadoop authentication method is set to [SIMPLE], but a Kerberos principal is specified. Continuing with [KERBEROS] authentication.
[2017-10-03T11:51:58,320][WARN ][o.e.r.RepositoriesService] [master-HOSTNAME] failed to create repository [hdfs][REPOSITORY]
java.security.AccessControlException: access denied ("java.security.SecurityPermission" "insertProvider.SaslPlainServer")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
	at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
	at java.lang.SecurityManager.checkSecurityAccess(SecurityManager.java:1759) ~[?:1.8.0_121]
	at java.security.Security.checkInsertProvider(Security.java:862) ~[?:1.8.0_121]
	at java.security.Security.insertProviderAt(Security.java:359) ~[?:1.8.0_121]
	at java.security.Security.addProvider(Security.java:403) ~[?:1.8.0_121]
	at org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:181) ~[?:?]
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:581) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
	at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:140) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:107) ~[?:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) ~[?:?]
	at com.sun.proxy.$Proxy34.mkdirs(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2525) ~[?:?]
	at org.apache.hadoop.fs.Hdfs.mkdir(Hdfs.java:311) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:738) ~[?:?]
	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:734) ~[?:?]
	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) ~[?:?]
	at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:741) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:65) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.lambda$execute$0(HdfsBlobStore.java:132) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
	at java.security.AccessController.doPrivileged(AccessController.java:713) ~[?:1.8.0_121]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.execute(HdfsBlobStore.java:129) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.mkdirs(HdfsBlobStore.java:62) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsBlobStore.<init>(HdfsBlobStore.java:55) ~[?:?]
	at org.elasticsearch.repositories.hdfs.HdfsRepository.doStart(HdfsRepository.java:116) ~[?:?]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.repositories.RepositoriesService.applyClusterState(RepositoriesService.java:303) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.2.jar:5.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.2.jar:5.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
	Suppressed: java.security.AccessControlException: access denied ("java.security.SecurityPermission" "insertProvider.SaslPlainServer")
		at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
		at java.security.AccessControlContext.checkPermission2(AccessControlContext.java:538) ~[?:1.8.0_121]
		at java.security.AccessControlContext.checkPermission(AccessControlContext.java:481) ~[?:1.8.0_121]
		at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
		at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
		at java.lang.SecurityManager.checkSecurityAccess(SecurityManager.java:1759) ~[?:1.8.0_121]
		at java.security.Security.checkInsertProvider(Security.java:865) ~[?:1.8.0_121]
		at java.security.Security.insertProviderAt(Security.java:359) ~[?:1.8.0_121]
		at java.security.Security.addProvider(Security.java:403) ~[?:1.8.0_121]
		at org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:181) ~[?:?]
		at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:581) ~[?:?]
		at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
		at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
		at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
		at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:140) ~[?:?]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:107) ~[?:?]
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) ~[?:?]
		at com.sun.proxy.$Proxy34.mkdirs(Unknown Source) ~[?:?]
		at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2525) ~[?:?]
		at org.apache.hadoop.fs.Hdfs.mkdir(Hdfs.java:311) ~[?:?]
		at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:738) ~[?:?]
		at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:734) ~[?:?]
		at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) ~[?:?]
		at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:741) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:65) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore$2.run(HdfsBlobStore.java:62) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore.lambda$execute$0(HdfsBlobStore.java:132) ~[?:?]
		at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
		at java.security.AccessController.doPrivileged(AccessController.java:713) ~[?:1.8.0_121]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore.execute(HdfsBlobStore.java:129) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore.mkdirs(HdfsBlobStore.java:62) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsBlobStore.<init>(HdfsBlobStore.java:55) ~[?:?]
		at org.elasticsearch.repositories.hdfs.HdfsRepository.doStart(HdfsRepository.java:116) ~[?:?]
		at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.repositories.RepositoriesService.applyClusterState(RepositoriesService.java:303) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.2.jar:5.6.2]
		at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.2.jar:5.6.2]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
		at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 3, 2017

Contributor

This relates to #26513 in that the work required is probably very similar.

Contributor

risdenk commented Oct 3, 2017

This relates to #26513 in that the work required is probably very similar.

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 3, 2017

Contributor

@jbaiera - FYI on this since you looked at #26513

Contributor

risdenk commented Oct 3, 2017

@jbaiera - FYI on this since you looked at #26513

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 3, 2017

Contributor

My first hunch from looking through the code previously is it relates to:

https://github.com/elastic/elasticsearch/blob/5.6/plugins/repository-hdfs/src/main/java/org/elasticsearch/repositories/hdfs/HdfsSecurityContext.java#L63

It looks like the permissions are added to the policy file (and being properly picked up) but not being passed in the context.

I don't have a test case for this right now but can look it into it later this week.

Contributor

risdenk commented Oct 3, 2017

My first hunch from looking through the code previously is it relates to:

https://github.com/elastic/elasticsearch/blob/5.6/plugins/repository-hdfs/src/main/java/org/elasticsearch/repositories/hdfs/HdfsSecurityContext.java#L63

It looks like the permissions are added to the policy file (and being properly picked up) but not being passed in the context.

I don't have a test case for this right now but can look it into it later this week.

@risdenk risdenk changed the title from java.lang.SecurityException: access denied ("java.lang.reflect.ReflectPermission" "suppressAccessChecks") for :Plugin Repository HDFS to java.lang.SecurityException: "putProviderProperty.SaslPlainServer" and "insertProvider.SaslPlainServer" for :Plugin Repository HDFS Oct 3, 2017

@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Oct 3, 2017

Contributor

Hi again @risdenk. Thanks for opening this issue.

Since HDFS takes a lot of liberties in regards to what permissions it has during operation (forking processes, reflective calls, access to user principals), we try to limit the permissions it has during regular operations to be a subset of all the permissions in the policy file (which includes all permissions it needs during the client's first set up).

While we do have them included in the policy file by default, the putProviderProperty.SaslPlainServer and insertProvider.SaslPlainServer permissions were considered unneeded for regular client operations after the client was created during testing. Thus these permissions were removed from the list of restricted permissions for client operations. Could you share the HDFS properties used on this issue as well as any information you find while attempting to get a clean reproduction? We'll need a use case that reproduces the errors in order to get a PR up to fix them.

Contributor

jbaiera commented Oct 3, 2017

Hi again @risdenk. Thanks for opening this issue.

Since HDFS takes a lot of liberties in regards to what permissions it has during operation (forking processes, reflective calls, access to user principals), we try to limit the permissions it has during regular operations to be a subset of all the permissions in the policy file (which includes all permissions it needs during the client's first set up).

While we do have them included in the policy file by default, the putProviderProperty.SaslPlainServer and insertProvider.SaslPlainServer permissions were considered unneeded for regular client operations after the client was created during testing. Thus these permissions were removed from the list of restricted permissions for client operations. Could you share the HDFS properties used on this issue as well as any information you find while attempting to get a clean reproduction? We'll need a use case that reproduces the errors in order to get a PR up to fix them.

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 5, 2017

Contributor

Yea I'll try to get to reproducing this cleanly, but likely won't be till next week. I will most likely focus on 6.0.0 to reproduce since that environment has no one on it (and also is closest to master to make PRs easier).

My hunch in case this gives any ideas. It looks like the error is with multiple repositories from a quick glance at the logs. The first repository is initialized just fine. The second (and third) are the ones that have the initialization issue. I haven't looked closer as to why this could be. From the last time I looked at the tests, they were all single repository though. Maybe the second repository initialization takes a different path.

Contributor

risdenk commented Oct 5, 2017

Yea I'll try to get to reproducing this cleanly, but likely won't be till next week. I will most likely focus on 6.0.0 to reproduce since that environment has no one on it (and also is closest to master to make PRs easier).

My hunch in case this gives any ideas. It looks like the error is with multiple repositories from a quick glance at the logs. The first repository is initialized just fine. The second (and third) are the ones that have the initialization issue. I haven't looked closer as to why this could be. From the last time I looked at the tests, they were all single repository though. Maybe the second repository initialization takes a different path.

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 6, 2017

Contributor

So I have a new hunch from reviewing the stack trace in more detail. I think this has to do with HDFS NameNode high availability. The code being executed is to create a failover proxy between two NameNodes. I don't think this is tested in the test framework since only a single NameNode is used. I am going to try switching the active NameNode in the cluster and see if that makes a difference as well.

The following part of the stacktrace is the same in both cases. It didn't appear in #26513.

...
at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
...

I haven't gotten this to reproduce cleanly yet outside of the environment though.

Contributor

risdenk commented Oct 6, 2017

So I have a new hunch from reviewing the stack trace in more detail. I think this has to do with HDFS NameNode high availability. The code being executed is to create a failover proxy between two NameNodes. I don't think this is tested in the test framework since only a single NameNode is used. I am going to try switching the active NameNode in the cluster and see if that makes a difference as well.

The following part of the stacktrace is the same in both cases. It didn't appear in #26513.

...
at org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343) ~[?:?]
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:170) ~[?:?]
at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:67) ~[?:?]
at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:151) ~[?:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.failover(RetryInvocationHandler.java:221) ~[?:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processRetryInfo(RetryInvocationHandler.java:147) ~[?:?]
...

I haven't gotten this to reproduce cleanly yet outside of the environment though.

@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Oct 6, 2017

Contributor

@risdenk That would make sense - After perusing the failover code, I found that when creating an HA-enabled client the HDFS Client code does not perform any operations or create any connections. Instead, the place where all that initialization would normally occur in a non-HA case is chopped off and done lazily when the user code initiates the first operation.

Quick question: Are you using this HA repository as a read-only repository like in #26513 ? I'm wondering if it's read only and the lack of a verification step is causing it to leave the HDFS client uninitialized.

Contributor

jbaiera commented Oct 6, 2017

@risdenk That would make sense - After perusing the failover code, I found that when creating an HA-enabled client the HDFS Client code does not perform any operations or create any connections. Instead, the place where all that initialization would normally occur in a non-HA case is chopped off and done lazily when the user code initiates the first operation.

Quick question: Are you using this HA repository as a read-only repository like in #26513 ? I'm wondering if it's read only and the lack of a verification step is causing it to leave the HDFS client uninitialized.

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 6, 2017

Contributor

@jbaiera - Yes all of our HDFS repository for 5.x and 6.x are readonly right now for testing (prevent someone from making a snapshot). We use 2.x to write to the HDFS repositories right now. We will switch from readonly to read/write for 5.x when we finish deployments.

Contributor

risdenk commented Oct 6, 2017

@jbaiera - Yes all of our HDFS repository for 5.x and 6.x are readonly right now for testing (prevent someone from making a snapshot). We use 2.x to write to the HDFS repositories right now. We will switch from readonly to read/write for 5.x when we finish deployments.

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 7, 2017

Contributor

I was able to reproduce this with a nice portable Docker implementation with NameNode HA. It reproduces with 5.6.2 and 6.0.0-rc1. It doesn't look like this has anything to do with readonly. The failure occurs with a read/write repository. The NameNode HA state is important. The error occurs when the first NameNode contacted is in Standby mode and must failover to the active NameNode. This is lazily done after the repository is setup. I haven't tested this but would be curious to see if the additional permissions are needed later if the NameNode HA state were to switch after initial client setup.

Here is the repo with everything needed to reproduce.

https://github.com/risdenk/elasticsearch_hdfs_kerberos_testing

The master branch has 5.6.2 and the 6.0.0-rc1 branch has 6.0.0-rc1.

The logs from the Travis test run if you don't want to set this up locally is here:

https://travis-ci.org/risdenk/elasticsearch_hdfs_kerberos_testing/branches

I haven't looked into how much effort it would be to make the Vagrant test framework for the HDFS secure tests support NameNode HA. It was a pain to get Docker and Kerberos to play nicely with hostnames.

Contributor

risdenk commented Oct 7, 2017

I was able to reproduce this with a nice portable Docker implementation with NameNode HA. It reproduces with 5.6.2 and 6.0.0-rc1. It doesn't look like this has anything to do with readonly. The failure occurs with a read/write repository. The NameNode HA state is important. The error occurs when the first NameNode contacted is in Standby mode and must failover to the active NameNode. This is lazily done after the repository is setup. I haven't tested this but would be curious to see if the additional permissions are needed later if the NameNode HA state were to switch after initial client setup.

Here is the repo with everything needed to reproduce.

https://github.com/risdenk/elasticsearch_hdfs_kerberos_testing

The master branch has 5.6.2 and the 6.0.0-rc1 branch has 6.0.0-rc1.

The logs from the Travis test run if you don't want to set this up locally is here:

https://travis-ci.org/risdenk/elasticsearch_hdfs_kerberos_testing/branches

I haven't looked into how much effort it would be to make the Vagrant test framework for the HDFS secure tests support NameNode HA. It was a pain to get Docker and Kerberos to play nicely with hostnames.

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 8, 2017

Contributor

Looks like adding the following to MiniHDFS.java might help. Would need to address the fixed ports still. It would be nice to able to specify the ports dynamically in the yml files for the rest-api-spec tests.

MiniDFSNNTopology miniDFSNNTopology = MiniDFSNNTopology.simpleHATopology();
builder.nnTopology(miniDFSNNTopology);
Contributor

risdenk commented Oct 8, 2017

Looks like adding the following to MiniHDFS.java might help. Would need to address the fixed ports still. It would be nice to able to specify the ports dynamically in the yml files for the rest-api-spec tests.

MiniDFSNNTopology miniDFSNNTopology = MiniDFSNNTopology.simpleHATopology();
builder.nnTopology(miniDFSNNTopology);
@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Oct 9, 2017

Contributor

Thanks for the footwork toward getting a reproduction. I think you might have hit the nail on the head - If the repository already exists and the leading namenode steps down, allowing a different one to take it's place as leader, the ensuing connection failover would require a whole new client to be initialized underneath the failover manager.

I'm working on getting this all to play nice in the test framework. I don't like the idea of throwing away our current permission rendering in all cases. If a user isn't using secured HDFS, there's no reason to permit any kerberos operations on the client at that time. In the event that a user IS using secured HDFS, we want to make sure that only the logged in user can initiate kerberos connections, and so on.

Most likely what we'll end up doing is sniff the client configuration for HA settings, and if we find any, we'll open up the rendered permissions only in the case of a client failover to allow for the connection to be re-established.

Contributor

jbaiera commented Oct 9, 2017

Thanks for the footwork toward getting a reproduction. I think you might have hit the nail on the head - If the repository already exists and the leading namenode steps down, allowing a different one to take it's place as leader, the ensuing connection failover would require a whole new client to be initialized underneath the failover manager.

I'm working on getting this all to play nice in the test framework. I don't like the idea of throwing away our current permission rendering in all cases. If a user isn't using secured HDFS, there's no reason to permit any kerberos operations on the client at that time. In the event that a user IS using secured HDFS, we want to make sure that only the logged in user can initiate kerberos connections, and so on.

Most likely what we'll end up doing is sniff the client configuration for HA settings, and if we find any, we'll open up the rendered permissions only in the case of a client failover to allow for the connection to be re-established.

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 10, 2017

Contributor

@jbaiera - Sounds good. Thanks for digging in further. I think I have the minimum required configs necessary to make HDFS HA work (in the docker-compose.yml) and for an Elasticsearch HDFS repository to use HDFS HA in run_test.sh under the variable HDFS_CONFIGS. Let me know if you need anything else to help test this out.

Contributor

risdenk commented Oct 10, 2017

@jbaiera - Sounds good. Thanks for digging in further. I think I have the minimum required configs necessary to make HDFS HA work (in the docker-compose.yml) and for an Elasticsearch HDFS repository to use HDFS HA in run_test.sh under the variable HDFS_CONFIGS. Let me know if you need anything else to help test this out.

@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Oct 12, 2017

Contributor

@risdenk Wanted to check in here again to say that I have the HDFS fixture spinning up a topology of HA Namenodes in the integration test environment, and was able to get a very very crude reproduction of the error. The biggest challenge with this bug is finding a way to get the Namenode to failover automatically during the integration test run - as right now I'm pulling all the levers by hand. Once all of that is in place, I can start getting a PR together that fixes the issue.

Thanks again for all the work you've put into this. HDFS permission issues always prove to be tricky.

Contributor

jbaiera commented Oct 12, 2017

@risdenk Wanted to check in here again to say that I have the HDFS fixture spinning up a topology of HA Namenodes in the integration test environment, and was able to get a very very crude reproduction of the error. The biggest challenge with this bug is finding a way to get the Namenode to failover automatically during the integration test run - as right now I'm pulling all the levers by hand. Once all of that is in place, I can start getting a PR together that fixes the issue.

Thanks again for all the work you've put into this. HDFS permission issues always prove to be tricky.

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 13, 2017

Contributor

@jbaiera thanks for the update. If there is a branch/PR that you want me to look at let me know. I've been around the Hadoop stuff quite a bit.

Contributor

risdenk commented Oct 13, 2017

@jbaiera thanks for the update. If there is a branch/PR that you want me to look at let me know. I've been around the Hadoop stuff quite a bit.

@jbaiera jbaiera added the >bug label Oct 18, 2017

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 30, 2017

Contributor

@jbaiera - Any update here?

Contributor

risdenk commented Oct 30, 2017

@jbaiera - Any update here?

@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Oct 30, 2017

Contributor

Hi @risdenk, automating the HA integration tests has been fairly difficult in terms of keeping them compliant with Elasticsearch's testing conventions. The good news is that I'm about 95% complete with the testing changes; Just need to pass a few more build-level conventions tests around naming and api usage. I'm hoping to have a preliminary PR up for review this week - but the changes to the testing infra are pretty verbose. We'll be testing HA HDFS (secured and unsecured) alongside the regular HDFS tests going forward. I will link back to this issue when the PR is available.

Contributor

jbaiera commented Oct 30, 2017

Hi @risdenk, automating the HA integration tests has been fairly difficult in terms of keeping them compliant with Elasticsearch's testing conventions. The good news is that I'm about 95% complete with the testing changes; Just need to pass a few more build-level conventions tests around naming and api usage. I'm hoping to have a preliminary PR up for review this week - but the changes to the testing infra are pretty verbose. We'll be testing HA HDFS (secured and unsecured) alongside the regular HDFS tests going forward. I will link back to this issue when the PR is available.

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Oct 30, 2017

Contributor

Thanks for the update @jbaiera

Contributor

risdenk commented Oct 30, 2017

Thanks for the update @jbaiera

@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Oct 31, 2017

Contributor

@risdenk I have a PR for this up now at #27196

Contributor

jbaiera commented Oct 31, 2017

@risdenk I have a PR for this up now at #27196

@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Dec 4, 2017

Contributor

This is fixed with #27196

Contributor

jbaiera commented Dec 4, 2017

This is fixed with #27196

@jbaiera jbaiera closed this Dec 4, 2017

@risdenk

This comment has been minimized.

Show comment
Hide comment
@risdenk

risdenk Dec 5, 2017

Contributor

Thanks @jbaiera!

Contributor

risdenk commented Dec 5, 2017

Thanks @jbaiera!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment