Spring data redis, multi-threading issue with Jedis

[deepakpol]

Hi,

I am getting into this issue while using Jedis with spring data redis integration. I am setting the configuration to use pool, testOnBorrow as well as testOnReturn true.

I am using Jedis pool, and have verified that spring ensures the resources are returned to the pool, by calling returnResource() and returnBrokenResource in case of exceptions. Still facing this issue. I have a heavily multi-threaded environment.

Jedis version - 2.6.0
Spring data redis version - 1.4.2

Error stack -

Thread 1:
[ERROR] [03/01/2015 07:05:32.044] [events-system-akka.actor.default-dispatcher-2281] [akka://events-system/user/$YN/$b/$b/$b] java.lang.Long cannot be cast to java.util.List
java.lang.ClassCastException: java.lang.Long cannot be cast to java.util.List
at redis.clients.jedis.Connection.getRawObjectMultiBulkReply(Connection.java:230)
at redis.clients.jedis.Connection.getObjectMultiBulkReply(Connection.java:236)
at redis.clients.jedis.BinaryJedis.zscan(BinaryJedis.java:3608)
at org.springframework.data.redis.connection.jedis.JedisConnection$3.doScan(JedisConnection.java:2998)
at org.springframework.data.redis.core.KeyBoundCursor.doScan(KeyBoundCursor.java:39)
at org.springframework.data.redis.core.ScanCursor.scan(ScanCursor.java:85)
at org.springframework.data.redis.core.ScanCursor.hasNext(ScanCursor.java:168)
at org.springframework.data.redis.core.ConvertingCursor.hasNext(ConvertingCursor.java:56)
...
application specific stack trace
...

Thread 2:

[ERROR] [03/01/2015 07:03:07.295] [events-system-akka.actor.default-dispatcher-2273] [akka://events-system/user/$VN/$b/$b/$b] Unknown redis exception; nested exception is java.lang.ClassCastException: [B cannot be cast to java.lang.Long
org.springframework.data.redis.RedisSystemException: Unknown redis exception; nested exception is java.lang.ClassCastException: [B cannot be cast to java.lang.Long
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.getFallback(FallbackExceptionTranslationStrategy.java:48)
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:38)
at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:195)
at org.springframework.data.redis.connection.jedis.JedisConnection.zRem(JedisConnection.java:2321)
at org.springframework.data.redis.core.DefaultZSetOperations$19.doInRedis(DefaultZSetOperations.java:283)
at org.springframework.data.redis.core.DefaultZSetOperations$19.doInRedis(DefaultZSetOperations.java:280)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:190)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:85)
at org.springframework.data.redis.core.DefaultZSetOperations.remove(DefaultZSetOperations.java:280)
...
application specific stack trace
...
Caused by: java.lang.ClassCastException: [B cannot be cast to java.lang.Long
at redis.clients.jedis.Connection.getIntegerReply(Connection.java:210)
at redis.clients.jedis.BinaryJedis.zrem(BinaryJedis.java:1624)
at org.springframework.data.redis.connection.jedis.JedisConnection.zRem(JedisConnection.java:2319)
... 21 more

[HeartSaVioR]

Seems like its InputStream was flawed.
Could you provide small code snippet or project that would reproduce current issue?

And I ask you a favor of updating spring-data-redis to latest and try again to confirm it's still an issue for current version(s).

[HeartSaVioR]

Is this error showed alone? Or you received other Exception first?

[deepakpol]

I am using the latest spring-data-redis version, any reason that should matter (has any fix gone related to this)?

I don't get any other errors before or after, this is the full stack trace for two different instances (except my application specific call stack).

I will try and provide some code snippet which doesn't give specifics of the application.

BTW, just to add to the question, I am using multiple spring RedisTemplates for serializing/deserializing different models. I have now configured different spring RedisTemplates with their own JedisConnectionFactory so that the jedis pool is created per factory, but this didn't solve the issue. I was always using a pool, and added testOnBorrow as well as testOnReturn true recently, but didn't help

I had posted a question on stackoverflow, where I have posted my spring configuration -
http://stackoverflow.com/questions/28814502/spring-data-redis-multi-threading-issue-with-jedis

[deepakpol]

Here is a snippet of code that gives this exception quite often. This could be accessed for execution by multiple threads -

    Cursor<ZSetOperations.TypedTuple<DomainObject>> cursor = null;

    cursor = redisObjectTemplate.opsForZSet().scan(KEY_PREFIX, scanOptions);

    List<DomainObject> objectsToBeDeleted = new ArrayList<>();
    while (cursor != null && cursor.hasNext()){
        objectsToBeDeleted.add(cursor.next().getValue())
    }
    //Delete Objects to be deleted from redis

I am also suspecting that the scan command is causing the synchronization issues since it is not executed atomically by spring, instead it starts with cursorId 0 and iterates until again the cursorId is returned as 0, so I am planning to change it to -

    try {
        lock.tryLock();
        cursor = redisObjectTemplate.opsForZSet().scan(KEY_PREFIX, scanOptions);
    } finally {
        lock.unlock();
    }

Let me know if this can help

Another snippet that might get executed in parallel which increments various counters in sequence, and also gives the exception sometimes -

redisTemplate.opsForValue().increment(COUNTER1_KEY_PREFIX , 1);
...
redisTemplate.opsForValue().increment(COUNTER2_KEY_PREFIX , 1);
...
redisTemplate.opsForValue().increment(COUNTER3_KEY_PREFIX , 1);

Hope this helps

[HeartSaVioR]

@deepakpol Could you also post this issue to Spring-data-redis?
https://jira.spring.io/browse/DATAREDIS
Actually it seems to be specific issue on Spring-data-redis which I don't have an idea. Sorry!
I will taking a look when I have some times.

[marcosnils]

@deepakpol any updates about this?

[tonivade]

Hi!

I have the same problem outside Spring.

There are several threads subscribed to differents subjects and when I try to kill those threads, the thread throws this exception:

Exception in thread "RediListenerThread-localhost:10001-alive" java.lang.ClassCastException: [B cannot be cast to java.util.List
    at redis.clients.jedis.Connection.getRawObjectMultiBulkReply(Connection.java:227)
    at redis.clients.jedis.JedisPubSub.process(JedisPubSub.java:108)
    at redis.clients.jedis.JedisPubSub.proceed(JedisPubSub.java:102)
    at redis.clients.jedis.Jedis.subscribe(Jedis.java:2496)
    at com.indra.davinci.common.cache.connector.redis.RedisListenerThread.run(RedisListenerThread.java:74)

We are currently using 2.6.2 version of jedis.

[marcosnils]

@tonivade any chance you can share a snippet or sample code that reproduces the problem?

[tonivade]

Sure!

We try to interrupt the thread because we don't needed it anymore.
If we call only listener.unsubscribe() method, other threads like this were still alive.
If we call jedis.quit() also, the thread stops, but this causes the ClassCastException.
We managed to solve the problem calling only jedis.stop() instead, this method closed the socket and these threads died gracefully.

    import java.util.logging.Level;
    import java.util.logging.Logger;

    import redis.clients.jedis.Jedis;
    import redis.clients.jedis.JedisPubSub;

    public class RedisListenerThread extends Thread {

        private static final Logger LOGGER = Logger.getLogger(RedisListenerThread.class.getName());

        private Jedis jedis;

        private JedisPubSub listener;

        private String channel;

        public RedisListenerThread(JedisPubSub listener, Jedis jedis, String channel) {
            super("RediListenerThread-" + jedis.getClient().getHost() + ":" + jedis.getClient().getPort() + "-" + channel);
            this.listener = listener;
            this.jedis = jedis;
            this.channel = channel;
        }

        @Override
        public void run() {
            try {
                // blocking operation (socket read)
                jedis.subscribe(listener, channel);
            } catch (Exception e) {
                LOGGER.log(Level.SEVERE, "listener error: " + getName(), e);
            }

            LOGGER.log(Level.INFO, "thread terminated: {0}", getName());
        }

        public void quit() {
            try {
                if (listener.isSubscribed()) {
                    listener.unsubscribe(channel);
                }
            } catch (Exception e) {
                LOGGER.log(Level.SEVERE, "listener error: " + getName(), e);
            }

            try {
                if (jedis.isConnected()) {
                    jedis.quit();
                }
            } catch (Exception e) {
                LOGGER.log(Level.SEVERE, "listener error: " + getName(), e);
            }
        }

    }

[marcosnils]

@tonivade seems like the issue is solved. Can you close the issue?

[srdoshi]

@deepakpol Hey Deepak were you able to solve the : java.lang.ClassCastException ? We are also using multithreading and getting these exceptions . We are using JedisPool

[chinmaym7430]

I am getting this exception when I am trying to scan a large number of keys in Redis. The problem starts when the number of keys exceeds 10k. I am using Spring Data Redis 1.7.2 and Jedis 2.8.1. I have set use-Pool to be true. Please, can someone help me solve this problem?

[marcosnils]

I am getting this exception when I am trying to scan a large number of keys in Redis. The problem starts when the number of keys exceeds 10k. I am using Spring Data Redis 1.7.2 and Jedis 2.8.1. I have set use-Pool to be true. Please, can someone help me solve this problem?

@mp911de is this a spring-data-redis issue?

[mp911de]

Not quite sure. Spring Data Redis uses pooling (not necessarily for older versions) to server a dedicated connection for each thread. I'd need a reproducible test case. some self-contained application class should do the job, see https://gist.github.com/mp911de/73d4b95b04a66eae084395beb25bd499 for a template.

[marcosnils]

@chinmaym7430 can you provide reproducible steps to get the error?
i.e:

1 - Start redis with X conf and Load redis with 20k keys
2 - Execute this code configured this way with spring-data-redis
3 - See it blow.

Otherwise it's a bit difficult to guess what might be happening.

[deepakpol]

Update on the issue - I did find what seemed to be causing the issue in my case - The framework that I was using to manage pool by default used ForkJoinPool which employs work-stealing. I changed it to use fixed thread pool which resolved the issue.

@srdoshi @marcosnils @chinmaym7430 apologies for delays in response... somehow the thread skipped my attention.

[marcosnils]

@deepakpol thx for coming back to update the issue.

[chinmaym7430]

Sorry for the delayed response. Here are the steps to reproduce the error:

1. Set up the spring configuration as denoted below:
   
2. Use the following code for searching keys by pattern(hscan):

@Override
    public List<Object> findByPattern(String pattern) {
        List<Object> listOfObjects = new ArrayList<>();
        Cursor<Entry<Object, Object>> cursorMap = redisTemplate.boundHashOps(SNAPSHOT_MODULE_KEY)
                .scan(ScanOptions.scanOptions().match(pattern).count(100).build());
        if (cursorMap != null) {
            while (cursorMap.hasNext()) {
                listOfObjects.add(cursorMap.next().getKey());
            }
            return listOfObjects;
        } else {
            return null;
        }
    }

1. Use the following code to call the scan the db with a pattern:

if (StringUtils.isNotBlank(logicalServerId)) {
    snapshotPattern = "*:"
    + moduleApplicationId.concat(":").concat(moduleName).concat(":").concat(logicalServerId);
            } else {
                snapshotPattern = "*:" + moduleApplicationId.concat(":").concat(moduleName);
            }
    List<Object> listOfSnapshotModulesKeys = snapshotRepo.findByPattern(snapshotPattern);

Here, the values of the different components of the pattern are being populated with some logic mentioned somewhere else.

When I execute the code, I get the error after sometime. The exact point where the error occurs can't be pin pointed. Please bear in mind that the error occurs only when the number of keys in redis db is upwards of 10k. Below that, everything runs fine.

NOTE: I have bypassed this problem by setting the usePool argument as 'False'. But I would still like to know why the problem occurs and how do I solve it.

[marcosnils]

@mp911de I'm not familiarized with spring-data-redis, would you mind checking if this happens to you too?

[mp911de]

That's a bug inside of Spring Data Redis related to connection synchronization. I created DATAREDIS-531 to track the issue. I think this ticket can be closed.

[marcosnils]

It's already closed :D. Thx @mp911de