Getting exception please suggest

[mukul4u2005]

com.hazelcast.nio.serialization.HazelcastSerializationException: info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.KryoException: Buffer underflow

[jerrinot]

@mukul4u2005: can you describe your setup a bit? do you have a reproducer?

[mukul4u2005]

I am using subzero-all-0.6 jar , It is working fine on my local machine but when I am deploying on production the deserialization is not working , The serialization is not giving any issue but when I am doing get operation on Map in hazelcast it is throwing this exception .

[jerrinot]

do you have subzero registered on all members and clients?

[mukul4u2005]

I am deploying hazelcast as embeded of war file and doing programmatic configuration as:

SubZero.useForClasses(config, ActionLogDetails.class, ActionLogResponse.class, ServerDetails.class);

So there are two tomcat instance and making cluster there. Some time i got same exception like during migration and not I am getting this exception when I am doing get operation on Map.

[jerrinot]

can you post a full stacktrace?

[mukul4u2005]

com.hazelcast.nio.serialization.HazelcastSerializationException: info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.KryoException: Buffer underflow.
Serialization trace:
envid (com.db.gto.dap.model.AllocationInstance)
allocationInstanceList (com.db.gto.dap.model.AllocationDetails)
    at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:61)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:178)
    at com.hazelcast.map.impl.query.QueryResultIterator.next(QueryResultIterator.java:59)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.db.gto.dap.service.DapTimeLineCalculationServiceImpl.filterDapInstancesByInstance(DapTimeLineCalculationServiceImpl.java:274)
    at com.db.gto.dap.service.DapTimeLineCalculationServiceImpl.getAllocationDataTableStatistics(DapTimeLineCalculationServiceImpl.java:112)
    at com.db.gto.dap.service.DapTimeLineCalculationServiceImpl.getTimeLineCalculation(DapTimeLineCalculationServiceImpl.java:47)
    at com.db.gto.dap.view.DapTimelineView.generateOverviewResponse(DapTimelineView.java:60)
    at com.db.gto.dap.controller.DapTimlineController.getDapTimeLineForPost(DapTimlineController.java:50)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:969)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:871)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:845)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.db.gto.commons.filter.AbstractFilter.doFilter(AbstractFilter.java:147)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.db.gto.commons.filter.SimpleCorsFilter.doFilter(SimpleCorsFilter.java:19)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:220)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:218)
    at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:333)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:442)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1082)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:623)
    at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2454)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.KryoException: Buffer underflow.
Serialization trace:
envid (com.db.gto.dap.model.AllocationInstance)
allocationInstanceList (com.db.gto.dap.model.AllocationDetails)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.io.Input.require(Input.java:199)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:373)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:831)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:780)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:540)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:540)
    at info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:709)
    at info.jerrinot.subzero.internal.strategy.TypedKryoStrategy.readObject(TypedKryoStrategy.java:24)
    at info.jerrinot.subzero.internal.strategy.KryoStrategy.read(KryoStrategy.java:76)
    at info.jerrinot.subzero.Serializer.read(Serializer.java:40)
    at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:46)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:172)
    ... 61 more

[jerrinot]

any chance you could share your domain classes? you can obfustate them, but it's hard to tell anything from the log.

[mukul4u2005]

public class AllocationDetails {

private List<AllocationInstance> allocationInstanceList;
private String atype;
private String aid;
private String dbscp;
private String name;
private String dbcrit;

getter-setter

and

public class AllocationInstance {

private String jstar;
private String regions;
private String jv;
private String tst;
private String sect;
private String nid;
private String envid;
private String dapUnits;
private String jds;
private String vefuk;
private String type;
private String masInScope;

getter-setter

[jerrinot]

can you try to create a reproducer? if there is a serialization issue then it should be possible to create a local reproducer - just start multiple Hazelcast instances in a single JVM.

[mukul4u2005]

hmmm I will try to create but I found on kryo website support some people facing similar exception because they did not have output.flush(); after writeObject method.

@OverRide
public void write(ObjectDataOutput out, Test object) throws IOException {
Output output = new Output((OutputStream) out);
getKryo().writeObject(output, object);
output.flush();

}

Is that can be reason. Mean in Subzero we have to mention sumwhere

[jerrinot]

SubZero calls the flush, see

subzero/subzero-core/src/main/java/info/jerrinot/subzero/internal/strategy/KryoStrategy.java

Line 70 in 5f80142

output.flush();

[mukul4u2005]

Ok I am trying to reproduce but for your information there are some exception in log files showing during replication as:

An exception occurred while executing migration operation com.hazelcast.map.impl.operation.MapReplicationOperation{serviceName='hz:impl:mapService', identityHash=270795521, partitionId=0, replicaIndex=0, callId=0, invocationTime=-1 (1970-01-01 00:59:59.999), waitTimeout=-1, callTimeout=9223372036854775807}
com.hazelcast.nio.serialization.HazelcastSerializationException: info.jerrinot.subzero.relocated.com.esotericsoftware.kryo.KryoException: Buffer underflow.
Serialization trace:

[jerrinot]

@mukul4u2005: hi, do you have any uppdate/reproducer?

[mukul4u2005]

@jerrinot: Hi, That I tried to reproduce on my local it is working fine but it is failing in INT environment with same implementation. SO I am not sure what is happening.