On restart of a Java Batch job, deserialization fails when checkpoint objects contain array type fields.

[scottkurz]

As discovered on StackOverflow here the code using Java serialization of checkpoint objects fails during deserialization when the checkpoint objects include array type fields.

Note that the StackOverflow question doesn't exactly conclusively show that array type fields were used there. Further testing shows this to be the case.

I'll copy the stack trace from the StackOverflow post, however:

[3/1/18 19:18:09:004 IST] [process partition0] com.ibm.ws.batch.JobLogger                                     CWWKY0030I: An exception occurred while running the step process.
com.ibm.jbatch.container.exception.BatchContainerRuntimeException: Failure in Read-Process-Write Loop
    at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.invokeChunk(ChunkStepControllerImpl.java:704)
    at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.invokeCoreStep(ChunkStepControllerImpl.java:795)
    at com.ibm.jbatch.container.controller.impl.BaseStepControllerImpl.execute(BaseStepControllerImpl.java:293)
    at com.ibm.jbatch.container.controller.impl.ExecutionTransitioner.doExecutionLoop(ExecutionTransitioner.java:118)
    at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.executeCoreTransitionLoop(WorkUnitThreadControllerImpl.java:93)
    at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.executeWorkUnit(WorkUnitThreadControllerImpl.java:155)
    at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl$AbstractControllerHelper.runExecutionOnThread(WorkUnitThreadControllerImpl.java:480)
    at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.runExecutionOnThread(WorkUnitThreadControllerImpl.java:89)
    at com.ibm.jbatch.container.util.BatchWorkUnit.run(BatchWorkUnit.java:117)
    at com.ibm.ws.context.service.serializable.ContextualRunnable.run(ContextualRunnable.java:79)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.ibm.jbatch.container.exception.BatchContainerRuntimeException: java.lang.ClassNotFoundException: [B
    at com.ibm.jbatch.container.util.TCCLObjectInputStream.resolveClass(TCCLObjectInputStream.java:40)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
    at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1671)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
    at java.io.ObjectInputStream.access$300(ObjectInputStream.java:208)
    at java.io.ObjectInputStream$GetFieldImpl.readFields(ObjectInputStream.java:2182)
    at java.io.ObjectInputStream.readFields(ObjectInputStream.java:543)
    at java.math.BigInteger.readObject(BigInteger.java:4406)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at java.util.HashMap.readObject(HashMap.java:1404)
    at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at java.util.HashMap.readObject(HashMap.java:1404)
    at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at com.ibm.jbatch.container.persistence.CheckpointData.<init>(CheckpointData.java:47)
    at com.ibm.jbatch.container.persistence.jpa.StepThreadInstanceEntity.getCheckpointData(StepThreadInstanceEntity.java:131)
    at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.openReaderAndWriter(ChunkStepControllerImpl.java:935)
    at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.invokeChunk(ChunkStepControllerImpl.java:599)
    ... 14 more
Caused by: java.lang.ClassNotFoundException: [B
    at com.ibm.ws.classloading.internal.UnifiedClassLoader.findClass(UnifiedClassLoader.java:119)
    at com.ibm.ws.classloading.internal.ThreadContextClassLoader.findClass(ThreadContextClassLoader.java:120)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at com.ibm.ws.classloading.internal.UnifiedClassLoader.loadClass0(UnifiedClassLoader.java:107)
    at com.ibm.ws.classloading.internal.UnifiedClassLoader$Delegation.loadClass(UnifiedClassLoader.java:78)
    at com.ibm.ws.classloading.internal.UnifiedClassLoader.loadClass(UnifiedClassLoader.java:102)
    at com.ibm.ws.classloading.internal.ThreadContextClassLoader.loadClass(ThreadContextClassLoader.java:136)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.ibm.jbatch.container.util.TCCLObjectInputStream.resolveClass(TCCLObjectInputStream.java:38)
    ... 57 more

A workaround would be to use ArrayList instead.

[scottkurz]

One other quick thing to fix:

Just noting that com.ibm.jbatch.container.persistence.CheckpointManager.checkpoint() rethrows its exceptions without logging them. Would be nice to add a bit more logging here.

[scottkurz]

Seems this problem might be dependent on the exact topology of the app-level classloaders. E.g. some reports of it happening in an Eclipse "loose config" but not in a dropins deployment (for a JAR project within WAR project app). Seems recreatable as well from the Maven+WDT-based BonusPayout app (which is Eclipse-based but only a single project).

Anyway, this isn't supposed to be clear things up..just to say if it doesn't get recreated, try another packaging.

[scottkurz]

Not sure if this is at all related... one more thing to look at:
https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/java_lang_ClassNotFoundException_Ljava_lang_String_after_updating_to_Java_8_0_5_5_or_later?lang=en

[scottkurz]

Very relevant discussion here:
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6516909

Perhaps we should be copying DeserializationObjectInputStream.java or using it directly.