Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hazelcast Client Near cache returns old value #8838

Closed
ohadbehore opened this issue Sep 4, 2016 · 22 comments
Closed

Hazelcast Client Near cache returns old value #8838

ohadbehore opened this issue Sep 4, 2016 · 22 comments
Labels
Milestone

Comments

@ohadbehore
Copy link

@ohadbehore ohadbehore commented Sep 4, 2016

I'm using Hazelcast 3.7.

I have an issue that an IMap configured with near-cache on Hazelcast client instance returns the old value (previous value that was stored in cache) after setting a new value using IMap=>submitToKey method with an entry processor.
From the Hazelcast documentation and local testing I saw that local near-cache should update when submitting an entry processor from the current client.
Is there a way I can configure a log (on Hazelcast client side) that will show all access to near cache (reads/writes)?
I want to make sure that no data is written to a specific key that I'm looking at, between the last update and the time the next read operation occurred (the one that returns the stale data).
The issue only occours when there is a load of about 50 (about 10:1 read:write ratio) requests per second on the Hazelcast client (Which is a weblogic application server).
My Weblogic Application that uses Hazelcast client is a multi threaded code, so I'm not eliminating the option that another thread might have updated the old value to the IMap again. But I need to check if this is the case.

I appriciate your help.
Ohad Behore

@ohadbehore
Copy link
Author

@ohadbehore ohadbehore commented Sep 4, 2016

I checked the same scenario without the near cache configuration and it works as designed. (All good)
The problem is that when I run a load twice bigger than the one described before the response times start to be unbearable (Very Big) for remote cache. And that is why I wanted to use the Near cache. To improve the performance.

@ahmetmircik
Copy link
Member

@ahmetmircik ahmetmircik commented Sep 5, 2016

Hi @ohadbehore,

Is the issue specific for map#submitToKey or for example, if you use map#put, do you still see it? Are all reads and writes happening from the same client which has near-cache? According to my tests (using submitToKey) when i read and write from the same near-cached client there shouldn't be any issue.

For the time being, no direct way to monitor near-cache read and writes. But you can listen updates on a key via EntryUpdatedListener.

Btw, is it possible to send a reproducer?

@ohadbehore
Copy link
Author

@ohadbehore ohadbehore commented Sep 5, 2016

Most Places I updated I use map#submitToKey but there is one place I update the map entries using map#executeOnEntries which includes SQL Predicate. Could this be the issue?

Regarding the EntryUpdatedListener is looking only at remote cache updates? Or does it look at the near cache?

@ahmetmircik
Copy link
Member

@ahmetmircik ahmetmircik commented Sep 5, 2016

Most Places I updated I use map#submitToKey but there is one place I update the map entries using map#executeOnEntries which includes SQL Predicate. Could this be the issue?

No it should be fine.

Regarding the EntryUpdatedListener is looking only at remote cache updates? Or does it look at the near cache?

EntryUpdatedListener can help you to listen all updated states of an entry in imap. You can't listen near-cache updates with that. But you can compare near-cached state of an entry with the actual state of it from imap.

From the Hazelcast documentation and local testing I saw that local near-cache should update when submitting an entry processor from the current client.

This is an excerpt from your first statement above
What did you mean with that? (Just trying to be sure that we are talking on same expectations. As you may know already, near-caches can be updated lately, they are not synchronously updated. )

@ohadbehore
Copy link
Author

@ohadbehore ohadbehore commented Sep 5, 2016

Regarding your comment:

From the Hazelcast documentation and local testing I saw that local near-cache should update when submitting an entry processor from the current client.

This is an excerpt from your first statement above
What did you mean with that? (Just trying to be sure that we are talking on same expectations. As you may know already, near-caches can be updated lately, they are not synchronously updated. )

From the following link (official Hazelcast documentation):
http://docs.hazelcast.org/docs/3.5/manual/html/jcache-nearcache.html

Invalidation is the process of removing an entry from the near cache since the entry is not valid anymore (its value is updated or it is removed from actual cache). Near cache invalidation happens asynchronously at the cluster level, but synchronously in real-time at the current node. This means when an entry is updated (explicitly or via entry processor) or removed (deleted explicitly or via entry processor, evicted, expired), it is invalidated from all near caches asynchronously within the whole cluster but updated/removed at/from the current node synchronously. Generally, whenever the state of an entry changes in the record store by updating its value or removing it, the invalidation event is sent for that entry.

On the current node (Hazelcast client) the near cache is updated immediately according to Hazelcast, as long as all updates are sent from the current node (explicitly or via entry processor).

@ahmetmircik
Copy link
Member

@ahmetmircik ahmetmircik commented Sep 5, 2016

From the following link (official Hazelcast documentation):
http://docs.hazelcast.org/docs/3.5/manual/html/jcache-nearcache.html

That page belongs to JCache near-cache implementation, The updated/removed at/from the current node synchronously part only valid for JCache near-cache, IMap near-cache doesn't have such a feature.

@ahmetmircik
Copy link
Member

@ahmetmircik ahmetmircik commented Sep 6, 2016

@ohadbehore where are we on this?

@ohadbehore
Copy link
Author

@ohadbehore ohadbehore commented Sep 6, 2016

The issue, I described above, occurs with Hazelcast client that has NearCache. From My understanding Hazelcast client that has a near cache, the near cache is configured as JCache. In the documentation it says :

NOTE: Near cache for JCache is only available for clients NOT servers.

I'm sorry if that was not clear, but I believe I did say that I was using Hazelcast client with near cache.

@ahmetmircik
Copy link
Member

@ahmetmircik ahmetmircik commented Sep 6, 2016

is it possible to create a reproducer?

@Donnerbart
Copy link
Contributor

@Donnerbart Donnerbart commented Sep 6, 2016

@ohadbehore You are misguided by that sentence in the documentation. We have a lot of different Near Cache scenarios and combinations of them:

  • IMap vs. JCache
  • Member vs. Client
  • On-heap (Hazelcast OpenSource) vs. HiDensity (Hazelcast Enterprise)

If you are working with an IMap data structure on the client, you are not using JCache. Please find the correct documentation here: http://docs.hazelcast.org/docs/3.7/manual/html-single/index.html#configuring-client-near-cache

Please find an explanation of the XML tags here: http://docs.hazelcast.org/docs/3.7/manual/html-single/index.html#creating-near-cache-for-map

There is also an explanation for the Near Cache invalidation in the context of IMap: http://docs.hazelcast.org/docs/3.7/manual/html-single/index.html#near-cache-invalidation

@ohadbehore
Copy link
Author

@ohadbehore ohadbehore commented Sep 7, 2016

Hi ahmetmircik and Donnerbart,
Thanks for the help.

But I'm trying to understand the issue here.
I need near cache for performance. Because using remote cache with Hazelcast client (IMAP without near cache) caused the response times of the Imap=>get and Imap=>submitToKey to rise to a very big and bad response times, over a second and even more than that.
There are a lot of updates and a lot more reads on my application regarding the part concerning the Hazelcast cache.
About 95% reads and 5% writes. and this could be around 1000 operations per second.

So if I use IMap with Near cache on Hazelcast client. How can I know that the near cache is updated?
I need a way to validate that the near cache has been updated.
I'm working on a copy of IMap with Near cache (on Hazelcast client) that I changed previously on the same Web server request (My Weblogic server application is using a Hazelcast client).
Should I use an entry listener to see that near cache is updated?

Again, I really appreciate the help.

@ahmetmircik
Copy link
Member

@ahmetmircik ahmetmircik commented Sep 7, 2016

You can check if latest nearCached value in sync with value on remote imap with something like this:

// Listen entry additions and updates on remote imap

private class TestListener implements EntryAddedListener<Integer, Integer>, EntryUpdatedListener<Integer, Integer> {

        ConcurrentHashMap<Integer, Integer> actualValueMap = new ConcurrentHashMap<Integer, Integer>();

        @Override
        public void entryAdded(EntryEvent<Integer, Integer> event) {
            update(event);
        }

        @Override
        public void entryUpdated(EntryEvent<Integer, Integer> event) {
            update(event);
        }

        private void update(EntryEvent<Integer, Integer> entryEvent) {
            Integer key = entryEvent.getKey();
            Integer value = entryEvent.getValue();

            actualValueMap.put(key, value);
        }

        public ConcurrentHashMap<Integer, Integer> getActualValueMap() {
            return actualValueMap;
        }
    }

// Check nearCachedValue against actualValue

        TestListener listener = new TestListener();
        map.addEntryListener(listener, true);

        Set<Integer> keys = map.keySet();
        for (Integer key : keys) {
            // this value possibly will come from local near-cache.
            Integer nearCachedValue = map.get(key);
            // this value will be the latest value from remote imap.
            Integer actualValue = listener.getActualValueMap().get(key);

            // nearCachedValues should be in sync with remote imap eventually, that's why we are  waiting in below loop
            boolean found = false;
            for (int i = 0; i < 3000; i++) {
                if (nearCachedValue.equals(actualValue)) {
                    found = true;
                    break;
                }
                parkNanos(MILLISECONDS.toNanos(10));
            }

            if (!found) {
                throw new AssertionError(format("key=%s, actualValue=%s, nearCachedValue=%s", key, actualValue, nearCachedValue));
            }
        }
@ohadbehore
Copy link
Author

@ohadbehore ohadbehore commented Sep 12, 2016

You can check if latest nearCached value in sync with value on remote imap with something like this:...

Ok, I checked it. And sometimes the Near cache is not accurate.
But I see that response times are inclining (on the scenario described above). Can the Entry listener you showed act as a Near Cache, because it is more accurate? Or is it not a good solution?
I'm trying to build a cache solution that has good response times and can replace my local Weblogic app cache?
I'm using Imap to store the Objects and entry processors to update them.
Any Ideas/recommendations to improve performance significantly?

@ohadbehore
Copy link
Author

@ohadbehore ohadbehore commented Sep 12, 2016

One more requirement, the data in the same Hazelcast client (Web-logic application) node sending the updates must be up to date.
I know this is a lot to ask, but this is our session management part of application and needs to be really fast and accurate.
Should I use Hazelcast as cache solution for a distributed cache and local cache solution?

@enesakar enesakar added the Team: Core label Nov 23, 2016
@enesakar enesakar added this to the 3.8 milestone Nov 23, 2016
@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Dec 12, 2016

@ahmetmircik: I assume this is fixed now?

@ohadbehore
Copy link
Author

@ohadbehore ohadbehore commented Dec 12, 2016

@ahmetmircik
Copy link
Member

@ahmetmircik ahmetmircik commented Dec 12, 2016

@ohadbehore, you did test with latest 3.7 which is 3.7.4?

@ohadbehore
Copy link
Author

@ohadbehore ohadbehore commented Dec 25, 2016

No I did not.
Is there any upgrade to the version that will solve this issue?
I think that there is an issue in modeling the problem.
We are trying to do session management with many different objects that update frequently.
The issue is that there are a lot of reads and a lot of writes in Hazelcast. The read write ratio is 95% read/ 5% write.
there may be even 10000 read/write operations in a minute.
And as the load test reaches that point the response times incline to a point where the server starts throwing errors.

@tombujok
Copy link
Contributor

@tombujok tombujok commented Jan 19, 2017

@ohadbehore It's hard to understand the real problem to me after reading this thread.
We would like to act on sth, but I don't really know where to start.

It somehow feels like two problems to me:

  1. Inaccurate near cache values Could you precisely describe your case and send a reproducer with a failing test, so that we may fully understand it?
  2. Increasing latency under high load Could you describe your case? We have a tool to test such high load scenarios and it would be good to reproduce it in our lab in order to help you with that.
@tombujok tombujok modified the milestones: 3.9, 3.8 Jan 19, 2017
@ahmetmircik
Copy link
Member

@ahmetmircik ahmetmircik commented Jul 6, 2017

Hi @ohadbehore, did you have a chance to retry with latest 3.7 or 3.8 releases? So far we couldn't reproduce the issue.

@ohadbehore
Copy link
Author

@ohadbehore ohadbehore commented Jul 6, 2017

@Donnerbart
Copy link
Contributor

@Donnerbart Donnerbart commented Jul 6, 2017

The ReplicatedMap doesn't replicate values to clients, just other members. So that might not solve your issue. A Hazelcast client never has local entries of a data structure, which are updated automatically. The Near Cache is just filled on client requests, not on any remote action of other nodes.

Nevertheless you have to understand that the Near Cache will always have a short period, where it returns a stale value, when the source value has been changed on another node. We cannot beat physics! And this is the trade-off of having any kind of local cache (it's super fast to read, but a remote update takes its time).

I'm also not sure if you fully understood the invalidation system and its tuning parameters. We overhauled the Near Cache documentation in Hazelcast 3.8. It's now a single chapter with all information in one place: http://docs.hazelcast.org/docs/3.8.3/manual/html-single/index.html#near-cache

Please have a close look at that (and don't confuse JCache with Near Cache, you're using IMap, so nothing with JCache will apply, e.g. local-update-policy is not available for IMap Near Caches).

I'm closing this issue, feel free to open it, if you can provide a reproducer to your original issue.

@Donnerbart Donnerbart closed this Jul 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.