Support re-start of HazelcastClient (after HazelcastInstanceNotActiveException)

[mzapletal]

Currently, clients may run into HazelcastInstanceNotActiveException and are shutting down in cases where this is not desired (e.g., reboot of the cluster, changing IP addresses ( #4349),...).
In such cases, it may be desirable to get the client from a shutdown state back to a connecting state.

While it is easy in plain Java applications to just re-create the Hazelcast client, this gets almost impossible (or needs a lot of workarounds) when working with dependency injection. For example, when working with Spring, one creates a HazelcastInstance (shielding a HazelcastClientProxy) in the container during startup of the application. When the HazelcastInstance bean then gets into the shutdown state it is virtually destroyed (even it would be possible to reconnect later on). As a consequence, the whole application is not able to recover from this error without a restart.

To overcome this issue, I would propose to support the restart of HazelcastClientProxy instances.

[gurbuzali]

an hackish way to restart client:

public static void restartHazelcastClient(HazelcastInstance clientInstance) {
        clientInstance.shutdown();
        ClassLoader tccl = Thread.currentThread().getContextClassLoader();
        try {
            Thread.currentThread().setContextClassLoader(HazelcastClient.class.getClassLoader());
            HazelcastClientProxy proxy =  ((HazelcastClientProxy) clientInstance);
            ClientConfig clientConfig = proxy.client.getClientConfig();
            HazelcastClientInstanceImpl client = new HazelcastClientInstanceImpl(clientConfig);
            client.start();
            OutOfMemoryErrorDispatcher.registerClient(client);
            proxy.client = client;
        } finally {
            Thread.currentThread().setContextClassLoader(tccl);
        }
    }

[mzapletal]

Ali, many thanks for the quick response and the provided code. We will try it out in our integration tests.

Any chance that such functionality will make it into the client?

[bwzhang2011]

@gurbuzali, is it the same way to restart the hazelcast itself. as one we shutdown the hazelcast server(instance) we should find way to restart that.

[bwzhang2011]

@gurbuzali, would you mind leaving more message for that ? If I were wrong to shutdown the hazelcastInstance from any kind of way. e.g from the press button of hazelcast mancenter, how could I restart such as I could catch the hazelcastInstanceNotActiveException and I could get the original configuration of hazelcast. how could I restart that or replace the bean in spring(if I use hazelcast-spring to create hazelcast instance). If you have any free time, would you mind leaving us more detail on how to restart hazelcast. for example, for the config hacked way to get from hazelcastInstanceProxy like this:
public static Config getHazelcastBoundConfig(HazelcastInstance hzInstance) {
if (hzInstance instanceof HazelcastInstanceProxy) {
HazelcastInstanceProxy hzInstanceProxy = (HazelcastInstanceProxy) hzInstance;

        Object originalHzInstance = BeanOperateHelper.getBeanDeclaredPropertySilent(hzInstanceProxy, "original");

        if (originalHzInstance instanceof HazelcastInstanceImpl) {
            HazelcastInstanceImpl hzInstanceImpl = (HazelcastInstanceImpl) originalHzInstance;

            return hzInstanceImpl.getConfig();
        }
    }

    return new Config();
}

but I was failed that I could not directly replace the bean associated with hazelcastInstance though I could get the config and try to new the hazelcastInstance again. I need such help and instructions from hz team.

[simpleusr]

@gurbuzali your provided solution does not seem to work at least with 3.8.3 release. Once a client switches to inactive state , client from HazelcastClientProxy is null

[sancar]

Hi @mzapletal , @simpleusr , @bwzhang2011
I would propose a different approach to the problem. Once a client is shutdown, we don't want to restart it. The shutdown state is a well-defined state for both hazelcast server&client, once it is shutdown there should be no resource/thread in use.

Instead of restarting the client, I would propose that configure it so that it will never shutdown. And always be re-usable.
This can be achieved by setting following properties
This will be the interval for a client to check if cluster is there once it is disconnected from cluster.
http://docs.hazelcast.org/docs/3.9/manual/html-single/index.html#setting-connection-timeout

And following will be the property to decide when to give up. If you want, you can set this practically infinite (Integer.MAX), or you can set a smaller value so that client will be closed for example after a week.
http://docs.hazelcast.org/docs/3.9/manual/html-single/index.html#setting-connection-attempt-limit

This proposal was not practical before 3.9. Because all operations on a disconnected client would block. But In 3.9, we have introduced async reconnect mode to give flexible behaviour in disconnected period.
For details you can see this doc.
http://docs.hazelcast.org/docs/3.9/manual/html-single/index.html#configuring-client-connection-strategy

[burakcelebi]

This feature is already available as explained by @sancar.
Please re-open the issue if you need further help.