Deserialization issue with Kryo 5 in parallel testing #798
Comments
|
I was able to write a test reproducing those issues here: https://github.com/altoo-ag/akka-kryo-serialization/blob/wip-parallel-serialization-issues-nvo/src/test/scala/io/altoo/akka/serialization/kryo/ParallelActorSystemSerializationTest.scala |
|
@nvollmar: Is any Kryo object or state shared between multiple threads? Since Kryo is not threadsafe, there is a lot that can go wrong in that case. The fact that the problem goes away when you switch to sequential tests indicates accidental multithreaded access. |
|
This was also our first thought, even though we use a pool to guard against multi-threaded access. We did not change anything in that regard when updating from Kryo 4 to 5, and the same code never had any issues with Kryo 4 in that regard. |
|
Interestingly the Now I'm really confused... |
Could it be that any other class is shared between threads? One of the resolvers or
The test code might have been incorrect before and the issue simply never manifested itself with Kryo 4.
Ok that's really strange. Could you try to replicate the problem in a minimal test-case without Akka? It could create a couple of threads and serialize/deserialize the same object your original test uses. If that test passes, there has to be some issue in your current test setup. |
I managed to write a test-case that reproduces this issue with plain Scala/Kryo: This reproduces the same serialization issue we observed: |
|
Thank you very much @nvollmar! I'll try to replicate the test in Java so we can get to the bottom of this. |
|
I rewrote your test in Java and it passes without problems: package com.esotericsoftware.kryo.serializers;
import static org.junit.Assert.*;
import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.io.Input;
import com.esotericsoftware.kryo.io.Output;
import com.esotericsoftware.kryo.util.DefaultClassResolver;
import com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy;
import com.esotericsoftware.kryo.util.MapReferenceResolver;
import com.esotericsoftware.kryo.util.Pool;
import java.io.Serializable;
import java.util.Objects;
import java.util.stream.IntStream;
import org.junit.Test;
import org.objenesis.strategy.StdInstantiatorStrategy;
public class ParallelSerializerTest {
private final Pool<Kryo> serializerPool = new Pool<Kryo>(true, false, 8) {
@Override
protected Kryo create () {
Kryo kryo = new Kryo(new DefaultClassResolver(), new MapReferenceResolver());
kryo.setRegistrationRequired(false);
kryo.setReferences(true);
DefaultInstantiatorStrategy instStrategy = (DefaultInstantiatorStrategy)kryo.getInstantiatorStrategy();
instStrategy.setFallbackInstantiatorStrategy(new StdInstantiatorStrategy());
kryo.setInstantiatorStrategy(instStrategy);
return kryo;
}
};
@Test
public void serializeAndDeserialize () {
IntStream.range(0, 1_000)
.parallel()
.forEach(i -> serializeDeserialize(new Sample1(new Option<>("test-" + i)), i));
}
private void serializeDeserialize (Sample1 sample, int i) {
System.out.println("Iteration: " + i);
Kryo sKryo = serializerPool.obtain();
Output out = new Output(4096, -1);
sKryo.writeClassAndObject(out, sample);
byte[] serialized = out.toBytes();
out.close();
serializerPool.free(sKryo);
Kryo dKryo = serializerPool.obtain();
Input in = new Input(serialized);
Sample1 result = (Sample1)dKryo.readClassAndObject(in);
in.close();
serializerPool.free(dKryo);
assertEquals(result, sample);
}
static final class Option<T> {
private final T value;
public Option (T value) {
this.value = value;
}
public T getValue () {
return value;
}
@Override
public boolean equals (Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Option<?> option = (Option<?>)o;
return Objects.equals(value, option.value);
}
@Override
public int hashCode () {
return Objects.hash(value);
}
}
static final class Sample1 implements Serializable {
private final Option<String> value;
Sample1 (Option<String> value) {
this.value = value;
}
public Option<String> getValue () {
return value;
}
@Override
public boolean equals (Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Sample1 sample1 = (Sample1)o;
return Objects.equals(value, sample1.value);
}
@Override
public int hashCode () {
return Objects.hash(value);
}
}
}There seems to be something special about the Scala |
|
Yes, the issue seems linked to the Scala To be noted, the Scala Did anything fundamentally change between Kryo 4 and 5 in in the area of |
|
I can also reproduce the issue with a simplified Supplemental: Now if I remove the |
|
Maybe this is helpful, the disassembled class files:
Note that a case class comes with a companion object:
|
|
Another finding (sorry for the flood of posts): The issue seems to manifest itself if a case class extends a generic class, removing the type parameter from |
|
@theigl I could come up with a pure Java example that fails serialization the same way: public abstract class JavaOption<T> implements Serializable {
}
class JavaSome<T> extends JavaOption<T> implements java.io.Serializable {
private final T value;
public JavaSome(T value) {
this.value = value;
}
public T value() {
return value;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
JavaSome<?> javaSome = (JavaSome<?>) o;
return Objects.equals(value, javaSome.value);
}
@Override
public int hashCode() {
return Objects.hash(value);
}
}
class JavaSample {
private final JavaOption<String> value;
public JavaSample(JavaOption<String> value) {
this.value = value;
}
public JavaOption<String> value() {
return value;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
JavaSample that = (JavaSample) o;
return Objects.equals(value, that.value);
}
@Override
public int hashCode() {
return Objects.hash(value);
}
} |
Could you please run the test with |
|
@nvollmar: Great work! I'll look into this! |
|
My Java test still passes with your sample class: package com.esotericsoftware.kryo.serializers;
import static org.junit.Assert.*;
import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.io.Input;
import com.esotericsoftware.kryo.io.Output;
import com.esotericsoftware.kryo.util.DefaultClassResolver;
import com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy;
import com.esotericsoftware.kryo.util.MapReferenceResolver;
import com.esotericsoftware.kryo.util.Pool;
import java.io.Serializable;
import java.util.Objects;
import java.util.stream.IntStream;
import org.junit.Test;
import org.objenesis.strategy.StdInstantiatorStrategy;
public class ParallelSerializerTest {
private final Pool<Kryo> serializerPool = new Pool<Kryo>(true, false, 8) {
@Override
protected Kryo create () {
Kryo kryo = new Kryo(new DefaultClassResolver(), new MapReferenceResolver());
kryo.setRegistrationRequired(false);
kryo.setReferences(true);
DefaultInstantiatorStrategy instStrategy = (DefaultInstantiatorStrategy)kryo.getInstantiatorStrategy();
instStrategy.setFallbackInstantiatorStrategy(new StdInstantiatorStrategy());
kryo.setInstantiatorStrategy(instStrategy);
return kryo;
}
};
@Test
public void serializeAndDeserialize () {
IntStream.range(0, 1_000)
.parallel()
.forEach(i -> serializeDeserialize(new JavaSample(new JavaSome<String>("test-" + i) {
}), i));
}
private void serializeDeserialize (JavaSample sample, int i) {
System.out.println("Iteration: " + i);
Kryo sKryo = serializerPool.obtain();
Output out = new Output(4096, -1);
sKryo.writeClassAndObject(out, sample);
byte[] serialized = out.toBytes();
out.close();
serializerPool.free(sKryo);
Kryo dKryo = serializerPool.obtain();
Input in = new Input(serialized);
JavaSample result = (JavaSample)dKryo.readClassAndObject(in);
in.close();
serializerPool.free(dKryo);
assertEquals(result, sample);
}
public abstract class JavaOption<T> implements Serializable {
}
class JavaSome<T> extends JavaOption<T> implements java.io.Serializable {
private final T value;
public JavaSome(T value) {
this.value = value;
}
public T value() {
return value;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
JavaSome<?> javaSome = (JavaSome<?>) o;
return Objects.equals(value, javaSome.value);
}
@Override
public int hashCode() {
return Objects.hash(value);
}
}
class JavaSample {
private final JavaOption<String> value;
public JavaSample(JavaOption<String> value) {
this.value = value;
}
public JavaOption<String> value() {
return value;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
JavaSample that = (JavaSample) o;
return Objects.equals(value, that.value);
}
@Override
public int hashCode() {
return Objects.hash(value);
}
}
}I'm not sure what the difference is between your Scala test and my test. What version of Kryo 5 are you using? What you could do is run your test against RC1-RC9. If it works on RC1, maybe we can pinpoint the change that introduced the issue. |
Looks like this would fix the issue, 20 successful test runs with it. |
Maybe you could add 3 more calls to
Was testing with 5.0.1 and 5.0.2
It already happens in RC1 |
That did it. I can reproduce the issue with 10_000 iterations like this: @Test
public void serializeAndDeserialize () {
IntStream.range(0, 10_000)
.parallel()
.forEach(it -> {
serializeDeserialize(new JavaSample(new JavaSome<>("test-" + it)), it);
serializeDeserialize(new JavaSample(new JavaSome<>("test-" + it)), it);
serializeDeserialize(new JavaSample(new JavaSome<>("test-" + it)), it);
serializeDeserialize(new JavaSample(new JavaSome<>("test-" + it)), it);
serializeDeserialize(new JavaSample(new JavaSome<>("test-" + it)), it);
});
}And it also passes when generics optimization is disabled. Kryo 5 enables generics optimization by default. In Kryo 4 this was an opt-in feature of Are you optimizing for speed or payload size? If speed is your main concern, I'd suggest you opt-out of generics optimization in your default configuration. I'll try to look into the root cause of this issue. Thanks again for your help! |
Apparently, multiple instances of a type variable can be created in highly concurrent environments.
Apparently, multiple instances of a type variable can be created in highly concurrent environments.
|
@nvollmar: OK, this was a PITA to debug, but I managed to find the issue. Apparently, in a highly concurrent environment, multiple instances of a kryo/src/com/esotericsoftware/kryo/util/DefaultGenerics.java Lines 139 to 144 in aaaff8b I created a PR that checks for object equality as well. |
|
Good catch 👍🏻 |
#798 Use equals to compare type variables
|
I just merged the PR. It would be great if you could run your original Scala test against the latest snapshot build, so we can be sure the issue is resolved. |
|
I can confirm that my test runs correct with 5.0.3-SNAPSHOT |
|
Great. I'll release 5.0.3 later this week. |
Since updating to Kryo 5 we are experience a weird deserialization failure when running tests parallel on multiple Akka actor systems in the same JVM. We could not reproduce this while running tests sequential nor did this ever happen with Kryo 4.
The issue occurs deserializing a plain
String, the field serializer tries to load a class by name with the content instead of just using that value. Any insight into what's going wrong would be very helpful.The text was updated successfully, but these errors were encountered: