Redisson RMap get response is slow

[shashibsingh]

I've a java application where I'm using Rmap to store and retrieve objects to/from Redis. The response time of map.put(...) is reasonable but map.get(...) is averaging 4000000.0 ns. I'm using FstCodec for serialization/deserialization. The application has many classes that do not provide no arg constructor and hence certain codecs can't be used.
I would like to know how to improve the map.get(...) response. When same object to go LinkedHashMap, the response time is in the range of 446.0 ns and I'm looking for a response time in that range.
Find attached my sample java application and config file.

RedissonIPWCTaskTester.java.txt
RedisConfig.json.txt

[jackygurui]

Can you show us your log from the code you provided?

[mrniko]

response time in 4 milliseconds is acceptable in your case. But I'm afraid it would be impossible to get response time in 446ns as you expect, because LinkedHashMap will always work faster than any network based in-memory solution.

[shashibsingh]

Here is the console output
Map population done
Map retrieval time is:9817.0
Key retrieval time is:4948399.0
Key retrieval time is:1.0683079E7
Key retrieval time is:3723906.0
Key retrieval time is:5642307.0
Key retrieval time is:4613715.0
Key retrieval time is:5433019.0
Key retrieval time is:2125906.0
Key retrieval time is:2769389.0
Key retrieval time is:1988016.0
Key retrieval time is:1586843.0
Key retrieval time is:1626559.0
Key retrieval time is:6298287.0
Key retrieval time is:2186149.0
Key retrieval time is:2500751.0
Key retrieval time is:2109395.0
Key retrieval time is:2070572.0
Key retrieval time is:2115642.0
Key retrieval time is:1727857.0
Key retrieval time is:1582828.0
Key retrieval time is:1288306.0

[shashibsingh]

Is it possible to find out what % of time is spent in object deserialization vs fetch from the mapo. Wanted to check if using alternate serialization could improve the performance, In our application cache is used heavily so even in small increment in response time is adding up considerably in overall response time.

[jackygurui]

You can try the RLocalCachedMap which can help you in this case.

[shashibsingh]

In that scenario, my application JVM should be able to handle all cached data - is this understanding correct? One of the reasons we're attempting to migrate from our home grown cache framework to Redisson/Redis is that we have large volume of data that needs to be cached and if it needs to be residing in application JVM, the application may slow down during garbage collection with increased heap size.

[jackygurui]

The RLocalCachedMap can be used to only cache some hot entries while leave the colder ones in Redis.

[shashibsingh]

Thanks for the response. I modified my tester program to read/write against RLocalCachedMap but don't see my of the performance gain. Here are the console output
Map population done
Map retrieval time is:166003.0
Key retrieval time is:2111626.0
Key retrieval time is:5112616.0
Key retrieval time is:5850704.0
Key retrieval time is:2712716.0
Key retrieval time is:2855960.0
Key retrieval time is:3088901.0
Key retrieval time is:5395982.0
Key retrieval time is:8943174.0
Key retrieval time is:1707775.0
Key retrieval time is:1877348.0
Key retrieval time is:2122336.0
Key retrieval time is:4868521.0
Key retrieval time is:2665415.0
Key retrieval time is:3696685.0
Key retrieval time is:2421765.0
Key retrieval time is:2531095.0
Key retrieval time is:4177735.0
Key retrieval time is:2605618.0
Key retrieval time is:4560166.0
Key retrieval time is:2121443.0

I also notice that the keys are being populated on Redis as well. Should data put into RLocalCachedMap be going to Redis server?

[jackygurui]

don't see my of the performance gain

It's because the data you requested is not cached locally. It a problem of a cache miss.

Should data put into RLocalCachedMap be going to Redis server?

It does. And also keeps some/all locally depending on your config

[shashibsingh]

Thanks again for your response. I've attached my sample program here. Could you please help me with appropriate config values.
RedissonIPWCTaskTester.java.txt

[shashibsingh]

I did some more iteration of test program by using RLocalCacheMap and RMap. Here are the findings.
Retrieval of 20 key based value retrieval from RLocalCacheMap is 8.81 times slower than sames operation on LinkedHashMap.
Retrieval of 20 key based value retrieval from RMap is 60.55 times slower than sames operation on LinkedHashMap.
Is this expected or I'm missing something in my configuration. The program is being run on my windows machine but performance is roughly same in Linux environment as well.
Appreciate your response.

[jackygurui]

I think you have misunderstood how RLocalCacheMap works:

1. RLocalCacheMap or any other Redis based objects are never ** meant to be faster or even as fast as LinkedHashMap.

2. RLocalCacheMap works by cache some/all data in JVM to avoid the network round trip, meaning you will only see the benefit when it has the data in cache.

3. When the requested data is not in the cache, a.k.a cache-miss, it will have to request the data with redis over the wire. Simply the higher the cache-miss rate, the slower the performance.

Looking at your code:

the producer is:

        for (int i = 0; i < 20; i++) {
            long id = i;
            RedissonIPWCTaskTester tester = new RedissonIPWCTaskTester();
            ComponentVO voTester = tester.getComponentVoObj(Long.valueOf(id), ComponentType.TASK, 1L, "Task-" + i);
            Map<String, ComponentVO> componentMap = populateMap(client, i, voTester, options, map);
            //System.out.println("Done:" + vo.getTitle());
        }
        System.out.println("Map population done");

and the consumer is:

    	for (int i =0; i < 20; i++){
    		double keyGetStart = System.nanoTime();
    		ComponentVO vo =(ComponentVO) map.get(Integer.toString(i));
     		double keyGetEnd = System.nanoTime();
    		System.out.println("Key retrieval time is:" + (keyGetEnd-keyGetStart));
        }

This shows the cache-miss rate is 100% which is the worst possible case.

If you only have 20 items in the map, and you only ever going to need each of them exactly once, there is no point use a local cache or near cache, however if you need to access some of them more than once, you will see the benefit.

Try the following code you should see the difference:

    	for (int i =0; i < 20; i++){
    		long keyGetStart = System.nanoTime();
    		ComponentVO vo =(ComponentVO) map.get(Integer.toString(i));
     		long keyGetEnd = System.nanoTime();
    		System.out.println("1st Key retrieval time is:" + (keyGetEnd-keyGetStart));
    		ComponentVO vo =(ComponentVO) map.get(Integer.toString(i));
    		ComponentVO vo =(ComponentVO) map.get(Integer.toString(i));
    		ComponentVO vo =(ComponentVO) map.get(Integer.toString(i));
    		ComponentVO vo =(ComponentVO) map.get(Integer.toString(i));
    		ComponentVO vo =(ComponentVO) map.get(Integer.toString(i));
     		long keyGetEnd1 = System.nanoTime();
    		System.out.println("Next 5 key retrieval time is:" + (keyGetEnd1-keyGetEnd)/5));
    		System.out.println("Average key retrieval time is:" + (keyGetEnd1-keyGetStart)/6));
        }

[shashibsingh]

Thanks for your response. The number 20 is arbitrarily chosen for this tester program. Inside application this number is running into 200000 to 500000 depending upon the environment. There are different kind of objects going into the cache and each object type goes into its own cache. Our LinkedHashMap based solution works fine with smaller data set but with increasing data , we're looking for something that doesn't impact application heap and hence garbage collection. Redis and Redisson was our obvious choice as we wanted to have our data residing in Redis and application works against that. Looks like with network traffic and serialization/deserialization of objects, it may not be most suitable for this particular application.
Thanks again for your time responding to the questions. Keep up the good work !!

[jackygurui]

The thing is a local cache / near cache and LinkedHashMap are in comparable in many ways:

LinkedHashMap is not thread-safe, it doesn't scale well, it can't be consistent across nodes. I think you have run into some of these issues before hence the looking around.

On the other hand the distributed local cache / near cache, the RLocalCachedMap in our case, is thread safe by nature, it can be scaled easily without much effort and more importantly, it provides a consistency policy that works across multiple node. Depending on how much JVM memory you are willing to provide, the performance can be leveraged to millions ops. Bear in mind while the data is still in JVM, there is no serialisation penalty to pay.

And I see you are also particularly interested in having a low latency while planning to keeping all the data in Redis. I think you might be interested in the Redisson PRO version. Redisson PRO is the enterprise version built on top of all the OS features. In addition it has a high performance engine as well as extra useful data types for enterprise environment. Based on our simple benchmark, we have observed 190k to 230k ops over the wire while the latency is within 1 ms.

[shashibsingh]

Hi @jackygurui - thanks for the response. As far as thread safety and synchronization is concerned, our LinkedHashMap based cache is handling it reasonably well using proprietary technology. And since all the nodes are caching java objects, time is not spent in serialization/de serialization. Having said that, we are interested in Redis based implementation to utilize other features of this framework including distributed cache thereby eliminating the need to continuously increase the heap of each of the nodes as data size increases.
I would love to try out the PRO version - sent a request via your site. Hope to get the PRO sometime soon as we need to make a decision to proceed with Redis based implementation in a day or two.

[jackygurui]

@shashibsingh No problem, I will have @mrniko to fast track the request for you.

[shashibsingh]

I got the license - thanks so much. Could you please point me to the jar file for pro. I'm getting following exception when introducing "registrationKey" in JSON config file
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "registrationKey" (class org.redisson.config.Config), not marked as ignorable (17 known properties: "useLinuxNativeEpoll", "eventLoopGroup", "keepPubSubOrder", "nettyThreads", "threads", "singleServerConfig", "sentinelServersConfig", "executor", "codec", "replicatedServersConfig", "clusterServersConfig", "masterSlaveServersConfig", "elasticacheServersConfig", "codecProvider", "lockWatchdogTimeout", "redissonReferenceEnabled", "resolverProvider"])
at [Source: (File); line: 27, column: 21] (through reference chain: org.redisson.config.Config["registrationKey"])

[jackygurui]

@shashibsingh Please use the sales@redisson.pro email address for fast tracked enterprise support.

[mrniko]

@shashibsingh Please make sure that you don't mix os version with pro in classpath.

[shashibsingh]

Thanks. That issue got resolved.
In order to reduce latency, should I be using RLocalCachedMap or Rmap also provides better response time in PRO version.

[jackygurui]

This is a question should really be translated into: Should I use a local cache / near cache (LC/NC) or not?

To consider if you want to use the LC/NC, there are a few questions should be asked:

1. Are you willing to provide some of your JVM memory in exchange for ultra fast data access.

2. If above is yes, do you think, based on your usage pattern, Are those data getting much more read request than write request?

3. If above is yes, do you think, within those read requests, you are likely to have the same data requested multiple times within a short time? I.e. A product detail information on a shopping site, the description of a hotel on a travel booking site, etc.

If your answer to all of above are yes then you should be using RLocalCachedMap other wise stick to the standard RMap.

[shashibsingh]

Thanks. My answer is 1 - Yes, 2 - Yes, 3- No. Based on your response I think I should stick with RMap.

[jackygurui]

OK, great! In which case the Redisson PRO is perfect for speed up the RMap

[shashibsingh]

I'm running the tests on my windows desktop. Will move the code to Linux and re run the test. Can I use this trial license on Window and Linux simultaneously or it is one connection at a time? Thx

[jackygurui]

you can run it on both platforms at the same time, but with Linux you can enable the epoll function which will give you better performance.

[shashibsingh]

great - let me check with my admin regarding epoll. Do you have any suggestion regarding map.put , map. fastPut and map.fastPutIfAbsent. In majority of the cases, we don't care about the previous value associated with a key.

[jackygurui]

fastPut will be better for you if you don't care about the previous value.

[shashibsingh]

good to know that - I guess I can try using async methods for put and see if that gets me some boost.

[jackygurui]

Be extra careful with async methods, since the nature of async, you will end up having a lot more future objects which can be a pain to the GC if it is not controlled. i.e. a for loop produces a million async calls might not be a good idea. Especially if the Redis side is not catching up with the demand, that would make some of those future objects live across a few GC generations which would require you to tweak JVM's GC params in order to get rid of them effectively.

Why not try the synced methods first and see the result, it should serve you as a guidance of what the PRO engine can do. Also try it against some real world use case, it is often to everyone's surprise that the PRO engine actually works better under load. It is tuned to be (nearly) linearly scaleable to a few hundreds of concurrent thread.

[shashibsingh]

Thanks for the suggestion - would you please point me to any documentation regarding epoll that you mentioned earlier. For me to be able to contact right admin, I must have little bit of background.

[shashibsingh]

Never mind - I think you're referring to https://en.wikipedia.org/wiki/Epoll

[jackygurui]

You mean the technical documentation for epoll? or the configuration document to have redisson use epoll?

[jackygurui]

I see you have found what you are looking for.

[shashibsingh]

Hi @jackygurui
Would you be able to provide exact function name and parameters for epoll ? My admins are asking for the details and I don't know much about this function,

[shashibsingh]

and just to be sure - this parameter needs to be changed on the server where Redis is installed not the app server of the application - correct?

[shashibsingh]

do we need to make any change in following config property
"useLinuxNativeEpoll": false

[mrniko]

@shashibsingh I'll try to answer. To start using epoll with Redisson you should set useLinuxNativeEpoll:true in configuration. You don't need to change anything on Redis server side.

[shashibsingh]

Hi @mrniko & @jackygurui
When we attempt to run the application on Linux, we're getting following exception.
From stack trace it appears to be classpath issue but we've verified by all possible means and there is no other jackson library in classpath. Only difference is that on my Windows env the application is exploded and on Linux its deployed as a war. Any thoughts?

Caused by: java.lang.NoSuchFieldError: ACCEPT_CASE_INSENSITIVE_PROPERTIES
at com.fasterxml.jackson.databind.deser.std.EnumDeserializer.createContextual(EnumDeserializer.java:146)
at com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:651)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:471)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
at com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:467)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:460)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4179)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
at org.redisson.config.ConfigSupport.fromJSON(ConfigSupport.java:201)
at org.redisson.config.Config.fromJSON(Config.java:639)

[mrniko]

It seems that you use different version of jackson lib in deployment environment. Please check classpath

[shashibsingh]

Hi there
Getting back from vacation and testing the PRO against production data. Not finding much difference in response time with PRO version. I'm inclined towards using RLocalCachedMap. Based on earlier response, I'm assuming Redisson would take care of synchronizing data across all nodes when RLocalCachedMap is used so as to avoid data inconsistency. Is my understanding correct? Please confirm.

[jackygurui]

@shashibsingh I am sorry you have not seen any differences in your use case with the PRO version, have you contacted @mrniko with your benchmark perhaps he can help you address the issues you are facing.

As to your another question, there is a config option called syncStrategy which you can specify to receive update when changed or invalidate the local cache upon receiving notifications from other nodes using the same RLocalCachedMap

[shashibsingh]

Hi @jackygurui Thanks for your response. I'll be contacting @mrniko with statistics. What I'm seeing is that with PRO, the application is reasonably fast 2nd login onwards with smaller DB. However, with production equivalent DB, it is slows down considerably. Wondering if I'm missing something in the implementation.

[shashibsingh]

Hi @jackygurui & @mrniko - Could you please point me to documentation for the following
RMap, RMapCache, RLocalCachedMap, RLocalCachedMapCache and provide recommendation as which one of these maps would provide fastest response.

[mrniko]

@shashibsingh Hi! All available documentation is on wiki page - https://github.com/redisson/redisson/wiki/7.-distributed-collections#714-map-data-partitioning

RLocalCachedMap - would fit best for fastest read operation response if you don't have requirements for ttl and maxIdleTime for map entry.
RClusteredLocalCachedMap - is the best option for both read and write operations.

[shashibsingh]

@mrniko Thanks for the response. After switching to RMapCache and increasing the cache size, I'm seeing the better performance. For comparison purpose, with RMap, to look-up a single key and return the associated java objets it is taking 1+ min whereas with RLocalCachedMap, it is taking 280 ms.
The LocalCachedMapOptions is configured as below.
LocalCachedMapOptions options = LocalCachedMapOptions.defaults()
.cacheSize(maxSize())
.evictionPolicy(EvictionPolicy.LRU)
.maxIdle(0, TimeUnit.SECONDS)
.timeToLive(0, TimeUnit.SECONDS)
.invalidateEntryOnChange(true);

With this behavior, got some questions and appreciate your response

1. Looking at the response time of RMap vs. RCahedMap where is the bottleneck - network round trip or serialization/deserialization.
2. When RCahedMap is used, does the map stores java objects natively or it is stored in a serialized form.
3. Is there any monitoring tool/api exist to find out the current statistics of local cache.
4. Any recommended approach to determine the optimal size for local cache.
5. Is there any performance difference b/w open source and PRO version when local cache map is used.
6. What is the recommended approach for synchronizing the nodes in a cluster.
7. Given the configuration I'm using, would eviction take place when max size is reached using LRU policy or some other algorithm going to be used for eviction.

[mrniko]

@shashibsingh

1+ min whereas with RLocalCachedMap, it is taking 280 ms

It seems you have very slow test and/or Redis instance.

Below are answers to your questions:

1. Bottleneck in network round trip
2. Java objects are stored as usual instances
3. Some metrics are available for RLocalCachedMap object https://github.com/redisson/redisson/wiki/14.-Integration-with-frameworks#metrics-per-rlocalcachedmap-object
4. It depends on use case and available memory
5. If value already in local cache there are no difference then, but if it's in Redis read operation is executed faster for multithreaded application
6. I recommend to use Redis Managed services like Azure Cache and AWS Elasticache. These services do that for you transparently.
7. LRU policy is used for eviction for the given configuration.