On a multi server setup, LDAP authentication doesn't work on all servers

[edmundoa]

On a setup with 2 Graylog servers, LDAP authentication doesn't work on all servers in certain circumstances. Trying to access an authenticated resource in one of the servers returns always 200, but on the other server 401.

I'm still looking into it and will update the ticket once I have more information.

[edmundoa]

Issue

On a multi server environment, LDAP authentication doesn't work all the time. In certain scenarios it is only possible to authenticate in one of the Graylog servers and not on both of them as it would be expected.

Environment

1. OpenLDAP server running on server A
2. Empty MongoDB running on server A
3. Empty ES running on server A and C
4. Two Graylog servers running on servers B (master) and C
5. Web interface running on server B

Steps to reproduce the issue

1. Start everything

2. Once both Graylog servers are in the cluster, go to the web interface

3. Login as admin user and set up LDAP

4. Log out

5. Try to log in with one of the users in LDAP

6. The login will only work on one of the servers. You can verify it by accessing an authenticated resource, in both servers and compare the results e.g:

   while true; do
       curl -XGET -s -i -u <user>:<password> http://<gl_serverB>:12900/system/throughput | grep 'HTTP'
       curl -XGET -s -i -u <user>:<password> http://<gl_serverC>:12900/system/throughput | grep 'HTTP'
       sleep 1
   done
   

Log from failed authentication attempt

2015-01-31 17:37:20,092 DEBUG: org.graylog2.security.realm.GraylogSimpleAccountRealm - Adding root account named admin, having all permissions
2015-01-31 17:37:20,095 DEBUG: org.apache.shiro.authc.pam.ModularRealmAuthenticator - Realm [org.graylog2.security.realm.SessionAuthenticator@a0f6783] does not support token org.apache.shiro.authc.UsernamePasswordToken - foobar, rememberM
e=false.  Skipping realm.
2015-01-31 17:37:20,096 DEBUG: org.apache.shiro.authc.pam.ModularRealmAuthenticator - Realm [org.graylog2.security.realm.AccessTokenAuthenticator@7954ce9d] does not support token org.apache.shiro.authc.UsernamePasswordToken - foobar, reme
mberMe=false.  Skipping realm.
2015-01-31 17:37:20,101 DEBUG: org.apache.directory.ldap.client.api.LdapConnectionConfig - found X509TrustManager sun.security.ssl.X509TrustManagerImpl@26efa8de
2015-01-31 17:37:20,101 DEBUG: org.apache.shiro.realm.AuthenticatingRealm - Looked up AuthenticationInfo [null] from doGetAuthenticationInfo
2015-01-31 17:37:20,101 DEBUG: org.apache.shiro.realm.AuthenticatingRealm - No AuthenticationInfo found for submitted AuthenticationToken [org.apache.shiro.authc.UsernamePasswordToken - foobar, rememberMe=false].  Returning null.
2015-01-31 17:37:20,101 DEBUG: org.graylog2.security.realm.PasswordAuthenticator - Retrieving authc info for user foobar
2015-01-31 17:37:20,102 DEBUG: org.graylog2.users.UserServiceImpl - Loading user foobar
2015-01-31 17:37:20,103 DEBUG: org.apache.shiro.realm.AuthenticatingRealm - Looked up AuthenticationInfo [null] from doGetAuthenticationInfo
2015-01-31 17:37:20,103 DEBUG: org.apache.shiro.realm.AuthenticatingRealm - No AuthenticationInfo found for submitted AuthenticationToken [org.apache.shiro.authc.UsernamePasswordToken - foobar, rememberMe=false].  Returning null.
2015-01-31 17:37:20,104 DEBUG: org.apache.shiro.realm.AuthenticatingRealm - Looked up AuthenticationInfo [null] from doGetAuthenticationInfo
2015-01-31 17:37:20,104 DEBUG: org.apache.shiro.realm.AuthenticatingRealm - No AuthenticationInfo found for submitted AuthenticationToken [org.apache.shiro.authc.UsernamePasswordToken - foobar, rememberMe=false].  Returning null.
2015-01-31 17:37:20,105 WARN : org.graylog2.rest.resources.system.SessionsResource - Unable to log in user foobar
org.apache.shiro.authc.AuthenticationException: No account information found for authentication token [org.apache.shiro.authc.UsernamePasswordToken - foobar, rememberMe=false] by this Authenticator instance.  Please check that it is configured correctly.
        at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:202)
        at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)
        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)
        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
        at org.graylog2.rest.resources.system.SessionsResource.newSession(SessionsResource.java:95)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:172)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2015-01-31 17:37:20,111 DEBUG: org.graylog2.rest.accesslog - 192.168.0.1 - [Sat Jan 31 17:37:20 UTC 2015] "POST system/sessions" graylog2-web/1.0.0-rc.1 (0b03650) 401 -1
2015-01-31 17:37:20,114 DEBUG: org.graylog2.jersey.container.netty.NettyContainer - Closing connection to /192.168.0.1:52711

[edmundoa]

Restarting the server with the problem fixes the issue (that's also why I couldn't reproduce it last time it happened), so it looks like we are not reloading the LDAP settings properly when they are updated.

[edmundoa]

After looking at the code, I think that the issue is there any time the LDAP settings change, not only on creation.