Session listeners are not called back in redis cluster

[sanketmeghani]

@rwinch We are using spring-session 1.2.0.RC1 and spring-data-redis 1.7.0.RC1. Somehow we are not getting the listener call back if we use redis cluster. It works perfect if we use a stand-alone redis server. I have uploaded a sample springSessionContext.xml.

On one of the nodes in redis cluster, we see "SUBSCRIBE" "__keyevent@:expired" "spring:session:event:created:" "__keyevent@*:del" command. So it does subscribe on one node. Should it subscribe to all nodes in the cluster? Or this is expected behavior?

Sample application code is available here.

[rwinch]

Thanks for the report!

cc @christophstrobl

[leehanwool]

I experienced same situation at tomcat 8 with spring-web-application(spring-data-redis 1.7.1.RELEASE) and redis-cluster(3.0.7).
When I stop tomcat, I received WARNING message like below, then tomcat going to hanging status, so I have to kill tomcat myself.(sometimes stop tomcat oneself after few minutes.)
would you please tell me about release plan with this issue?

[WARN] [RedisMessageListenerContainer$SubscriptionTask:864] : Unable to close connection. Subscription task still running
 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [redisMessageListenerContainer-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.net.SocketInputStream.socketRead0(Native Method)
 java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 java.net.SocketInputStream.read(SocketInputStream.java:170)
 java.net.SocketInputStream.read(SocketInputStream.java:141)
 java.net.SocketInputStream.read(SocketInputStream.java:127)
 redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:195)
 redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40)
 redis.clients.jedis.Protocol.process(Protocol.java:141)
 redis.clients.jedis.Protocol.read(Protocol.java:205)
 redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)
 redis.clients.jedis.Connection.getRawObjectMultiBulkReply(Connection.java:242)
 redis.clients.jedis.BinaryJedisPubSub.process(BinaryJedisPubSub.java:87)
 redis.clients.jedis.BinaryJedisPubSub.proceed(BinaryJedisPubSub.java:82)
 redis.clients.jedis.BinaryJedis.subscribe(BinaryJedis.java:2980)
 redis.clients.jedis.BinaryJedisCluster$150.execute(BinaryJedisCluster.java:1607)
 redis.clients.jedis.BinaryJedisCluster$150.execute(BinaryJedisCluster.java:1604)
 redis.clients.jedis.JedisClusterCommand.runWithAnyNode(JedisClusterCommand.java:87)
 redis.clients.jedis.BinaryJedisCluster.psubscribe(BinaryJedisCluster.java:1610)
 org.springframework.data.redis.connection.jedis.JedisClusterConnection.pSubscribe(JedisClusterConnection.java:2508)
 org.springframework.data.redis.listener.RedisMessageListenerContainer$SubscriptionTask.eventuallyPerformSubscription(RedisMessageListenerContainer.java:779)
 org.springframework.data.redis.listener.RedisMessageListenerContainer$SubscriptionTask.run(RedisMessageListenerContainer.java:746)
 java.lang.Thread.run(Thread.java:745)

[rwinch]

@leehanwool I haven't had a chance to look into this issue yet. Note that we also have a ticket for official Redis Cluster support that we need to get to. See #422

As for your StackTrace, could you please create a separate ticket for the unclosed thread?

[christophstrobl]

@rwinch @leehanwool created DATAREDIS-493 to check that.

[leehanwool]

@rwinch
Thank for reply.
I created new #482 ticket for Stacktrace.

[nirav-patel]

@rwinch @leehanwool @christophstrobl
I have done spring-session configuration with Redis by using Jedis Client (2.8.1 version) as mentioned in issue description by @sanketmeghani above. I found that internally when cluster is created, each node gets subscribed to patterns for event notifications (basically those are session creation, session expiration and session deletion events) via method

void redis.clients.jedis.BinaryJedisCluster.psubscribe(BinaryJedisPubSub jedisPubSub, byte[]... patterns)

Actual implementation of the method in BinaryJedisCluster class is as shown below:
Note in execute method, Jedis Connection is subscribed to patterns using subscribe method [i.e. connection.subscribe(jedisPubSub, patterns);].

@Override
  public void psubscribe(final BinaryJedisPubSub jedisPubSub, final byte[]... patterns) {
    new JedisClusterCommand<Integer>(connectionHandler, maxRedirections) {
      @Override
      public Integer execute(Jedis connection) {
        connection.subscribe(jedisPubSub, patterns);
        return 0;
      }
    }.runWithAnyNode();
  }

Here If I override this method with custom implementation and subscribe each Jedis Connection in execute method with psubscribe method instead of subscribe method [i.e., connection.psubscribe(jedisPubSub, patterns)], then I`m able to retrieve session notifications for creation, expiration and deletion activities performed by Redis Cluster.

@Override
  public void psubscribe(final BinaryJedisPubSub jedisPubSub, final byte[]... patterns) {
    new JedisClusterCommand<Integer>(connectionHandler, maxRedirections) {
      @Override
      public Integer execute(Jedis connection) {
        connection.psubscribe(jedisPubSub, patterns);
        return 0;
      }
    }.runWithAnyNode();
  }

In case of single Redis node implementation, Jedis Connection is subscribed to patterns using psubscribe method and not the subscribe method. So I applied the same logic for Redis Cluster use-case too and things are working fine.

I don`t know whether it is a Jedis issue or current implementation is correct for some purpose.
Is this a correct way to fix above issue and get session listeners call backs from Redis Cluster?

@rwinch @leehanwool @christophstrobl Please provide your feedback for this solution.

[rwinch]

@nirav-patel Thanks for the detailed writeup!

I think @christophstrobl will be able to provide better feedback on your solution

[leehanwool]

@nirav-patel , Thank you for advise!
I tried to use your solution in my case #482
It works perfectly. Tomcat no more going to hanging status, So I don't need kill tomcat anymore.
Thank you!!!

[christophstrobl]

thanks @nirav-patel for detailed information.

[HeartSaVioR]

Thanks all for finding missed spot for Jedis.

[rwinch]

Thanks for the reports and helping sort this out! I'm closing this as it appears the issue has been fixed in Jedis