Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hazelcast spring client-server recovery mechanism lacking #12128

Closed
apixandru opened this issue Jan 10, 2018 · 4 comments
Closed

Hazelcast spring client-server recovery mechanism lacking #12128

apixandru opened this issue Jan 10, 2018 · 4 comments

Comments

@apixandru
Copy link
Contributor

@apixandru apixandru commented Jan 10, 2018

Sample project available here:
https://github.com/apixandru/hazelcast-stuff

I can't seem to find a good way to deal with hazelcast running in client mode when the server restarts.

In the attached project you will find what I believe to be a correct configuration for jcache usage (and commented code to switch to spring the caching mechanism)

To simulate the problem

  1. start the server
  2. start the client
  3. visit the client http://localhost:8080/ - the connection with the server should be established now and data published to it
  4. restart the server
  5. visit the client again - it will fail because it can't find the cache.

With jcache, once the connection to the server node is lost, it does not matter if it's recovered, it will always fail with com.hazelcast.cache.CacheNotExistsException: Cache /hz/test is already destroyed or not created yet, on Member [192.168.0.206]:15000 - 8f6edf1e-c0a5-4f8d-b540-6ec2935c096b this

With spring cache, it looks like recovery is successful.
There is also a problem that once the hazelcast instance goes into recovery mode, all operations on it block. There appear to be only two ways to control this

  1. hazelcast.spring.cache.prop - not very useful, this only times out reads, writes still leave the application hanging
  2. hazelcast.client.invocation.timeout.seconds - looks like this times out any operation, the correct choice?

It appears that by far the best solution would be to simply let the HazelcastInstance die on connection to server lost and recreate it, however with spring integration this is not so great because the instance was already injected everywhere.

Is there an official way of dealing with this problem, am I missing something or am I simply using it wrong?

@apixandru
Copy link
Contributor Author

@apixandru apixandru commented Jan 11, 2018

Just for playing around, i wrote a small adapter that would clear the cache manager once the connectivity is lost. (to try and just recreate the client rather than hanging on to it)

https://github.com/apixandru/hazelcast-stuff/tree/recreate-hz
I just created a HazelcastInstance that delegates to the real HazelcastInstance. If the real HazelcastInstance is dead, we simply create a new one. The missing piece was a way to remove the stale references in the HazelcastCacheManager.
I added a lifecycle listener to the client and whenever the client died, it just cleared the cache cache. That way, when a new client comes online, it doesn't try to use the cached cache, but it simply creates a new one.

But this is something for the simple spring cache abstraction, jcache still won't work as it actually casts the HazelcastInstance to the actual implemenation class. It looks like it's tied up to an actual instance which doesn't allow for such flexibility.

@neilstevenson
Copy link
Contributor

@neilstevenson neilstevenson commented Jan 11, 2018

JCache 1.1 may handle this differently.

@neilstevenson
Copy link
Contributor

@neilstevenson neilstevenson commented Jan 13, 2018

Will check JCACHE 1.1

@neilstevenson
Copy link
Contributor

@neilstevenson neilstevenson commented Jan 16, 2018

JCache 1.1 doesn't make a difference, still gives "Cache /hz/test is already destroyed or not created yet".

The problem seems to be that the client has a proxy onto the ICache, but the ICache doesn't exist on the server side when the server is restarted.

A workaround is for a server side initializer to touch the ICache, creating it. Then the client's proxy is fine when the client reconnects.

When the client reconnects, it should drop/create the ICache proxies to ensure the ICache objects exist on the server side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

6 participants
You can’t perform that action at this time.