3.6.x client cannot obtain cache proxy from 3.7.x cluster

[vbekiaris]

To reproduce, have a 3.6 client connect on a 3.7 cluster and execute:

HazelcastInstance hz = HazelcastClient.newHazelcastClient();
CachingProvider cachingProvider = HazelcastClientCachingProvider.createCachingProvider(hz);
CacheManager cacheManager = cachingProvider.getCacheManager();
Cache<String,String> cache = cacheManager.getCache("test", String.class, String.class);

Client fails with:

Exception in thread "main" javax.cache.CacheException: Error opening URI [hazelcast]
    at com.hazelcast.cache.impl.AbstractHazelcastCachingProvider.getCacheManager(AbstractHazelcastCachingProvider.java:97)
    at com.hazelcast.cache.impl.AbstractHazelcastCachingProvider.getCacheManager(AbstractHazelcastCachingProvider.java:126)
    at CacheClientBootstrap.main(CacheClientBootstrap.java:33)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: com.hazelcast.cache.CacheNotExistsException: Couldn't find cache config with name pRef
    at com.hazelcast.cache.impl.AbstractCacheService.checkCacheSimpleConfig(AbstractCacheService.java:197)
    at com.hazelcast.cache.impl.AbstractCacheService.createDistributedObject(AbstractCacheService.java:177)
    at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.doCreateProxy(ProxyRegistry.java:177)
    at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.createProxy(ProxyRegistry.java:170)
    at com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.initializeDistributedObject(ProxyServiceImpl.java:133)
    at com.hazelcast.spi.impl.proxyservice.impl.operations.InitializeDistributedObjectOperation.run(InitializeDistributedObjectOperation.java:42)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:181)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:375)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:402)
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:283)
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:268)
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:232)
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:207)
    at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:59)
    at com.hazelcast.client.impl.protocol.task.AbstractInvocationMessageTask.processMessage(AbstractInvocationMessageTask.java:42)
    at com.hazelcast.client.impl.protocol.task.AbstractMessageTask.initializeAndProcessMessage(AbstractMessageTask.java:119)
    at com.hazelcast.client.impl.protocol.task.AbstractMessageTask.run(AbstractMessageTask.java:99)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
    at ------ submitted from ------.(Unknown Source)
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:111)
    at com.hazelcast.spi.impl.AbstractInvocationFuture$1.run(AbstractInvocationFuture.java:246)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
    at ------ End remote and begin local stack-trace ------.(Unknown Source)
    at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveResponse(ClientInvocationFuture.java:128)
    at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:95)
    at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:74)
    at com.hazelcast.client.spi.ProxyManager.initialize(ProxyManager.java:301)
    at com.hazelcast.client.spi.ProxyManager.initializeWithRetry(ProxyManager.java:257)
    at com.hazelcast.client.spi.ProxyManager.getOrCreateProxy(ProxyManager.java:238)
    at com.hazelcast.client.impl.HazelcastClientInstanceImpl.getDistributedObject(HazelcastClientInstanceImpl.java:511)
    at com.hazelcast.client.impl.HazelcastClientProxy.getDistributedObject(HazelcastClientProxy.java:241)
    at com.hazelcast.client.cache.impl.HazelcastClientCacheManager.<init>(HazelcastClientCacheManager.java:68)
    at com.hazelcast.client.cache.impl.HazelcastClientCachingProvider.createHazelcastCacheManager(HazelcastClientCachingProvider.java:67)
    at com.hazelcast.client.cache.impl.HazelcastClientCachingProvider.createHazelcastCacheManager(HazelcastClientCachingProvider.java:38)
    at com.hazelcast.cache.impl.AbstractHazelcastCachingProvider.getCacheManager(AbstractHazelcastCachingProvider.java:94)
    ... 7 more

[tombujok]

Managed to reproduce. Issues identified:

• setupRef initialization sequence was removed in 3.7
• changes in the config model, following fields added:

EvictionConfig:
protected String comparatorClassName;
protected EvictionPolicyComparator comparator;

CacheConfig:
private boolean disablePerEntryInvalidationEvents;

[ihsandemir]

Solution proposal:
In CacheCreateConfigMessageTask, prepareOperation method will try to do nodeEngine.toObject(parameters.cacheConfig) but if this fails it will use serializationService.toObject(parameters.cacheConfig, LegacyCacheConfig.class) to convert the config to Object.

[mdogan]

@ihsandemir & @asimarslan: Don't you think @tombujok's solution in #9010, casting ObjectDataInput to BufferObjectDataInput and checking for available data, is a good solution? It looks straightforward to me, no need to restore LegacyCacheConfig.

Just forget about this 👆 comment, @asimarslan reminded me why that solution doesn't work.

[ihsandemir]

There was a problem regarding the solution for the below case:

client 3.7.1 and server 3.7.2 case: During createCache request, the server is returning a cacheconfig object, our solution was to return the leagcycacheconfig object which works for 3.6 client but it does not work for 3.7 or 3.7.1 client and we have no way to differentiate the connected client is 3.6 or 3.7 version since the version info is added at 3.7.2.

After some discussions, we realized that if the customer is using JCache then at this moment they can not have the 3.6.x clients with with 3.7.x cluster due to the CacheConfig serialization problem.

Here is a possible solution:
By default the server returns the CacheConfig object if the client version can not be determined unless a compatibility property

    /**
     \* When this property is true, if the server can not determine the connected client version, it
     \* shall assume that it is of 3.6.x version client.
     */
    public static final HazelcastProperty COMPATIBILITY_3_6_CLIENT_ENABLED
            = new HazelcastProperty("hazelcast.compatibility.3.6_client", false);

is set to true. If this property is set, then it will return LegacyCacheConfig object.

Furthermore, a similar switch shall exist for the client:

    /**
     \* When this property is true, if the client can not know the server version it will assume that the server is version 3.6.x.
     */
    public static final HazelcastProperty COMPATIBILITY_3_6_SERVER_ENABLED
            = new HazelcastProperty("hazelcast.compatibility.3.6_server", false);

With this config when 3.7.3 client is talking to 3.6.x server it shall send LegacyCacheConfig in createCache request.