Hazelcast Client Near cache returns old value

[ohadbehore]

I'm using Hazelcast 3.7.

I have an issue that an IMap configured with near-cache on Hazelcast client instance returns the old value (previous value that was stored in cache) after setting a new value using IMap=>submitToKey method with an entry processor.
From the Hazelcast documentation and local testing I saw that local near-cache should update when submitting an entry processor from the current client.
Is there a way I can configure a log (on Hazelcast client side) that will show all access to near cache (reads/writes)?
I want to make sure that no data is written to a specific key that I'm looking at, between the last update and the time the next read operation occurred (the one that returns the stale data).
The issue only occours when there is a load of about 50 (about 10:1 read:write ratio) requests per second on the Hazelcast client (Which is a weblogic application server).
My Weblogic Application that uses Hazelcast client is a multi threaded code, so I'm not eliminating the option that another thread might have updated the old value to the IMap again. But I need to check if this is the case.

I appriciate your help.
Ohad Behore

[ohadbehore]

I checked the same scenario without the near cache configuration and it works as designed. (All good)
The problem is that when I run a load twice bigger than the one described before the response times start to be unbearable (Very Big) for remote cache. And that is why I wanted to use the Near cache. To improve the performance.

[ahmetmircik]

Hi @ohadbehore,

Is the issue specific for map#submitToKey or for example, if you use map#put, do you still see it? Are all reads and writes happening from the same client which has near-cache? According to my tests (using submitToKey) when i read and write from the same near-cached client there shouldn't be any issue.

For the time being, no direct way to monitor near-cache read and writes. But you can listen updates on a key via EntryUpdatedListener.

Btw, is it possible to send a reproducer?

[ohadbehore]

Most Places I updated I use map#submitToKey but there is one place I update the map entries using map#executeOnEntries which includes SQL Predicate. Could this be the issue?

Regarding the EntryUpdatedListener is looking only at remote cache updates? Or does it look at the near cache?

[ahmetmircik]

Most Places I updated I use map#submitToKey but there is one place I update the map entries using map#executeOnEntries which includes SQL Predicate. Could this be the issue?

No it should be fine.

Regarding the EntryUpdatedListener is looking only at remote cache updates? Or does it look at the near cache?

EntryUpdatedListener can help you to listen all updated states of an entry in imap. You can't listen near-cache updates with that. But you can compare near-cached state of an entry with the actual state of it from imap.

From the Hazelcast documentation and local testing I saw that local near-cache should update when submitting an entry processor from the current client.

This is an excerpt from your first statement above
What did you mean with that? (Just trying to be sure that we are talking on same expectations. As you may know already, near-caches can be updated lately, they are not synchronously updated. )

[ohadbehore]

Regarding your comment:

From the Hazelcast documentation and local testing I saw that local near-cache should update when submitting an entry processor from the current client.

This is an excerpt from your first statement above
What did you mean with that? (Just trying to be sure that we are talking on same expectations. As you may know already, near-caches can be updated lately, they are not synchronously updated. )

From the following link (official Hazelcast documentation):
http://docs.hazelcast.org/docs/3.5/manual/html/jcache-nearcache.html

Invalidation is the process of removing an entry from the near cache since the entry is not valid anymore (its value is updated or it is removed from actual cache). Near cache invalidation happens asynchronously at the cluster level, but synchronously in real-time at the current node. This means when an entry is updated (explicitly or via entry processor) or removed (deleted explicitly or via entry processor, evicted, expired), it is invalidated from all near caches asynchronously within the whole cluster but updated/removed at/from the current node synchronously. Generally, whenever the state of an entry changes in the record store by updating its value or removing it, the invalidation event is sent for that entry.

On the current node (Hazelcast client) the near cache is updated immediately according to Hazelcast, as long as all updates are sent from the current node (explicitly or via entry processor).

[ahmetmircik]

From the following link (official Hazelcast documentation):
http://docs.hazelcast.org/docs/3.5/manual/html/jcache-nearcache.html

That page belongs to JCache near-cache implementation, The updated/removed at/from the current node synchronously part only valid for JCache near-cache, IMap near-cache doesn't have such a feature.

[ahmetmircik]

@ohadbehore where are we on this?

[ohadbehore]

The issue, I described above, occurs with Hazelcast client that has NearCache. From My understanding Hazelcast client that has a near cache, the near cache is configured as JCache. In the documentation it says :

NOTE: Near cache for JCache is only available for clients NOT servers.

I'm sorry if that was not clear, but I believe I did say that I was using Hazelcast client with near cache.

[ahmetmircik]

is it possible to create a reproducer?

[Donnerbart]

@ohadbehore You are misguided by that sentence in the documentation. We have a lot of different Near Cache scenarios and combinations of them:

• IMap vs. JCache
• Member vs. Client
• On-heap (Hazelcast OpenSource) vs. HiDensity (Hazelcast Enterprise)

If you are working with an IMap data structure on the client, you are not using JCache. Please find the correct documentation here: http://docs.hazelcast.org/docs/3.7/manual/html-single/index.html#configuring-client-near-cache

Please find an explanation of the XML tags here: http://docs.hazelcast.org/docs/3.7/manual/html-single/index.html#creating-near-cache-for-map

There is also an explanation for the Near Cache invalidation in the context of IMap: http://docs.hazelcast.org/docs/3.7/manual/html-single/index.html#near-cache-invalidation

[ohadbehore]

Hi ahmetmircik and Donnerbart,
Thanks for the help.

But I'm trying to understand the issue here.
I need near cache for performance. Because using remote cache with Hazelcast client (IMAP without near cache) caused the response times of the Imap=>get and Imap=>submitToKey to rise to a very big and bad response times, over a second and even more than that.
There are a lot of updates and a lot more reads on my application regarding the part concerning the Hazelcast cache.
About 95% reads and 5% writes. and this could be around 1000 operations per second.

So if I use IMap with Near cache on Hazelcast client. How can I know that the near cache is updated?
I need a way to validate that the near cache has been updated.
I'm working on a copy of IMap with Near cache (on Hazelcast client) that I changed previously on the same Web server request (My Weblogic server application is using a Hazelcast client).
Should I use an entry listener to see that near cache is updated?

Again, I really appreciate the help.

[ahmetmircik]

You can check if latest nearCached value in sync with value on remote imap with something like this:

// Listen entry additions and updates on remote imap

private class TestListener implements EntryAddedListener<Integer, Integer>, EntryUpdatedListener<Integer, Integer> {

        ConcurrentHashMap<Integer, Integer> actualValueMap = new ConcurrentHashMap<Integer, Integer>();

        @Override
        public void entryAdded(EntryEvent<Integer, Integer> event) {
            update(event);
        }

        @Override
        public void entryUpdated(EntryEvent<Integer, Integer> event) {
            update(event);
        }

        private void update(EntryEvent<Integer, Integer> entryEvent) {
            Integer key = entryEvent.getKey();
            Integer value = entryEvent.getValue();

            actualValueMap.put(key, value);
        }

        public ConcurrentHashMap<Integer, Integer> getActualValueMap() {
            return actualValueMap;
        }
    }

// Check nearCachedValue against actualValue

        TestListener listener = new TestListener();
        map.addEntryListener(listener, true);

        Set<Integer> keys = map.keySet();
        for (Integer key : keys) {
            // this value possibly will come from local near-cache.
            Integer nearCachedValue = map.get(key);
            // this value will be the latest value from remote imap.
            Integer actualValue = listener.getActualValueMap().get(key);

            // nearCachedValues should be in sync with remote imap eventually, that's why we are  waiting in below loop
            boolean found = false;
            for (int i = 0; i < 3000; i++) {
                if (nearCachedValue.equals(actualValue)) {
                    found = true;
                    break;
                }
                parkNanos(MILLISECONDS.toNanos(10));
            }

            if (!found) {
                throw new AssertionError(format("key=%s, actualValue=%s, nearCachedValue=%s", key, actualValue, nearCachedValue));
            }
        }

[ohadbehore]

You can check if latest nearCached value in sync with value on remote imap with something like this:...

Ok, I checked it. And sometimes the Near cache is not accurate.
But I see that response times are inclining (on the scenario described above). Can the Entry listener you showed act as a Near Cache, because it is more accurate? Or is it not a good solution?
I'm trying to build a cache solution that has good response times and can replace my local Weblogic app cache?
I'm using Imap to store the Objects and entry processors to update them.
Any Ideas/recommendations to improve performance significantly?

[ohadbehore]

One more requirement, the data in the same Hazelcast client (Web-logic application) node sending the updates must be up to date.
I know this is a lot to ask, but this is our session management part of application and needs to be really fast and accurate.
Should I use Hazelcast as cache solution for a distributed cache and local cache solution?

[jerrinot]

@ahmetmircik: I assume this is fixed now?

[ohadbehore]

Unfortunately this did not work. בתאריך יום ב׳, 12 בדצמ׳ 2016, 14:52, מאת Jaromir Hamala ‏< notifications@github.com>:

…

@ahmetmircik <https://github.com/ahmetmircik>: I assume this is fixed now? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8838 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKNfXYhuPpcmVIPinYB2s32v1YpIfNVPks5rHUN1gaJpZM4J0iKh> .

[ahmetmircik]

@ohadbehore, you did test with latest 3.7 which is 3.7.4?

[ohadbehore]

No I did not.
Is there any upgrade to the version that will solve this issue?
I think that there is an issue in modeling the problem.
We are trying to do session management with many different objects that update frequently.
The issue is that there are a lot of reads and a lot of writes in Hazelcast. The read write ratio is 95% read/ 5% write.
there may be even 10000 read/write operations in a minute.
And as the load test reaches that point the response times incline to a point where the server starts throwing errors.

[tombujok]

@ohadbehore It's hard to understand the real problem to me after reading this thread.
We would like to act on sth, but I don't really know where to start.

It somehow feels like two problems to me:

1. Inaccurate near cache values Could you precisely describe your case and send a reproducer with a failing test, so that we may fully understand it?
2. Increasing latency under high load Could you describe your case? We have a tool to test such high load scenarios and it would be good to reproduce it in our lab in order to help you with that.

[ahmetmircik]

Hi @ohadbehore, did you have a chance to retry with latest 3.7 or 3.8 releases? So far we couldn't reproduce the issue.

[ohadbehore]

Hi, I think that multiple updates to cache are more fit to replicated map structure. Although we cannot get a guarantee for consistency as thee is in an Imap. I thin the issue was one of trying to load values that less than one millisecond ago have been updated. This will probably work only on replicated map structure. Thank you, I think you can close the bug. Ohad behore

…

On Thu, Jul 6, 2017, 2:38 PM ahmetmircik ***@***.***> wrote: Hi @ohadbehore <https://github.com/ohadbehore>, did you have a chance to retry with latest 3.7 or 3.8 releases? So far we couldn't reproduce the issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8838 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKNfXabAeeZAR1Jq9VJRV7VQX9dA0g7vks5sLMc8gaJpZM4J0iKh> .

[Donnerbart]

The ReplicatedMap doesn't replicate values to clients, just other members. So that might not solve your issue. A Hazelcast client never has local entries of a data structure, which are updated automatically. The Near Cache is just filled on client requests, not on any remote action of other nodes.

Nevertheless you have to understand that the Near Cache will always have a short period, where it returns a stale value, when the source value has been changed on another node. We cannot beat physics! And this is the trade-off of having any kind of local cache (it's super fast to read, but a remote update takes its time).

I'm also not sure if you fully understood the invalidation system and its tuning parameters. We overhauled the Near Cache documentation in Hazelcast 3.8. It's now a single chapter with all information in one place: http://docs.hazelcast.org/docs/3.8.3/manual/html-single/index.html#near-cache

Please have a close look at that (and don't confuse JCache with Near Cache, you're using IMap, so nothing with JCache will apply, e.g. local-update-policy is not available for IMap Near Caches).

I'm closing this issue, feel free to open it, if you can provide a reproducer to your original issue.