cannot start HZ cluster because of com.hazelcast.nio.serialization.HazelcastSerializationException

[dsukhoroslov]

HZ 3.7.
First node starts fine. when I start second node I get an exception:

2016-08-29 18:57:40.404 [hz.hzInstance.cached.thread-2] ERROR com.bagri.xdm.cache.hazelcast.task.schema.SchemaInitiator - initSchema.error; Error creating bean with name 'queryManager' defined in URL [file:../config/spring/cache-schema-context.xml]: Cannot resolve reference to bean 'xdmRepo' while setting bean property 'repository'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'xdmRepo' defined in URL [file:../config/spring/cache-schema-context.xml]: Error setting property values; nested exception is org.springframework.beans.PropertyBatchUpdateException; nested PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationException: Property 'txManagement' threw exception; nested exception is com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.hazelcast.concurrent.atomiclong.operations.AlterAndGetOperation'
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'queryManager' defined in URL [file:../config/spring/cache-schema-context.xml]: Cannot resolve reference to bean 'xdmRepo' while setting bean property 'repository'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'xdmRepo' defined in URL [file:../config/spring/cache-schema-context.xml]: Error setting property values; nested exception is org.springframework.beans.PropertyBatchUpdateException; nested PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationException: Property 'txManagement' threw exception; nested exception is com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.hazelcast.concurrent.atomiclong.operations.AlterAndGetOperation'
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1481) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1226) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at com.bagri.xdm.cache.hazelcast.task.schema.SchemaInitiator.call(SchemaInitiator.java:64) [bagri-server-hazelcast-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
    at com.bagri.xdm.cache.hazelcast.task.schema.SchemaInitiator.call(SchemaInitiator.java:29) [bagri-server-hazelcast-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_74]
    at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:241) [hazelcast-all-3.7.jar:3.7]
    at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:212) [hazelcast-all-3.7.jar:3.7]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_74]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_74]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-all-3.7.jar:3.7]
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) [hazelcast-all-3.7.jar:3.7]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'xdmRepo' defined in URL [file:../config/spring/cache-schema-context.xml]: Error setting property values; nested exception is org.springframework.beans.PropertyBatchUpdateException; nested PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationException: Property 'txManagement' threw exception; nested exception is com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.hazelcast.concurrent.atomiclong.operations.AlterAndGetOperation'
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1518) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1226) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    ... 22 common frames omitted
Caused by: org.springframework.beans.PropertyBatchUpdateException: Failed properties: Property 'txManagement' threw exception; nested exception is com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.hazelcast.concurrent.atomiclong.operations.AlterAndGetOperation'
    at org.springframework.beans.AbstractPropertyAccessor.setPropertyValues(AbstractPropertyAccessor.java:121) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.AbstractPropertyAccessor.setPropertyValues(AbstractPropertyAccessor.java:75) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1514) ~[spring-beans-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    ... 30 common frames omitted

All this stuff works properly in HZ 3.6. Please have a look.

Thanks, Denis.

[jerrinot]

Hi @dsukhoroslov,

thanks for the report. Can you share your config file?

[dsukhoroslov]

Hi @jerrinot,

Here we are: context

[dsukhoroslov]

@jerrinot

I just tried to run exactly same case on HZ 3.6.4 and got another exception in my code:

2016-08-30 14:49:43.808 [cached2] INFO  com.hazelcast.core.LifecycleService - [10.249.143.130]:10501 [default] [3.6.4] Address[10.249.143.130]:10501 is SHUTDOWN
2016-08-30 14:49:43.820 [cached2] ERROR com.bagri.xdm.cache.hazelcast.task.schema.SchemaInitiator - initSchema.error; Error creating bean with name 'queryManager' defined in URL [file:../config/spring/cache-schema-context.xml]: Cannot resolve reference to bean 'xdmRepo' while setting bean property 'repository'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'xdmRepo' defined in URL [file:../config/spring/cache-schema-context.xml]: Error setting property values; nested exception is org.springframework.beans.PropertyBatchUpdateException; nested PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationException: Property 'txManagement' threw exception; nested exception is com.hazelcast.nio.serialization.HazelcastSerializationException: java.io.NotSerializableException: com.bagri.xdm.client.hazelcast.impl.IdGeneratorImpl
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'queryManager' defined in URL [file:../config/spring/cache-schema-context.xml]: Cannot resolve reference to bean 'xdmRepo' while setting bean property 'repository'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'xdmRepo' defined in URL [file:../config/spring/cache-schema-context.xml]: Error setting property values; nested exception is org.springframework.beans.PropertyBatchUpdateException; nested PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationException: Property 'txManagement' threw exception; nested exception is com.hazelcast.nio.serialization.HazelcastSerializationException: java.io.NotSerializableException: com.bagri.xdm.client.hazelcast.impl.IdGeneratorImpl

Looks like it causes your SerializationException. So, leave it to me, I'll fix my code, then will try again with HZ 3.7.

Thanks, Denis

[dsukhoroslov]

Yep, it was a bug on my side. Have no idea how could I miss it.
Now tested on HZ 3.6.x and 3.7, everything works fine.

[jerrinot]

Hi @dsukhoroslov,

thanks for heads-up!