Deal with inner case classes #122
Comments
|
We could make a better version of the FieldsSerializer that works for case classes. These could be create at the first time the class is seen and could do that expensive check on outer being used the first time. |
|
Just ran into this issue – in my case it was throwing a |
|
We can't fix this example. That case class keeps a reference to test which
is the outer class. That's just the way scala compiles it. We could try to
determine that it is not needed by looking at all the bytecode and nulling
it out in that case, but that would be a fairly non-trivial to do, and no
one has yet taken it on.
Having people make the outer class serializable or removing the nesting is
what we recommend.
…On Tue, Aug 1, 2017 at 07:54 Bekir Oguz ***@***.***> wrote:
I see this issue is open for a very long time already, I also have the
same problem with FieldSerializer serializing case classes. The following
little unit test can reproduce this.
import com.twitter.chill.{Input, Output, ScalaKryoInstantiator}import org.scalatest.{FunSuite, Matchers}
class KryoSerializerSpec extends FunSuite with Matchers {
case class Test1(a: Int, b: String)
test("kryo can serialize/deserialize case classes") {
val buffer = new Array[Byte](4096)
val input = new Input(buffer)
val output = new Output(buffer)
val obj = Test1(5, "test1")
val kryo = (new ScalaKryoInstantiator).newKryo()
kryo.writeObject(output, obj)
kryo.readObject(input, classOf[Test1]) shouldBe obj
}
}
This is the tracktrace:
java.util.ConcurrentModificationException
Serialization trace:
classes (sun.misc.Launcher$AppClassLoader)
contextClassLoader (java.lang.Thread)
value (java.util.concurrent.atomic.AtomicReference)
atomic (org.scalatest.ConcurrentAlerter)
value (java.util.concurrent.atomic.AtomicReference)
atomicAlerter (org.scalatest.Engine)
org$scalatest$FunSuiteLike$$engine (com.ing.baker.runtime.actor.KryoSerializerSpec)
$outer (com.ing.baker.runtime.actor.KryoSerializerSpec$Test1)
com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
Serialization trace:
classes (sun.misc.Launcher$AppClassLoader)
contextClassLoader (java.lang.Thread)
value (java.util.concurrent.atomic.AtomicReference)
atomic (org.scalatest.ConcurrentAlerter)
value (java.util.concurrent.atomic.AtomicReference)
atomicAlerter (org.scalatest.Engine)
org$scalatest$FunSuiteLike$$engine (com.ing.baker.runtime.actor.KryoSerializerSpec)
$outer (com.ing.baker.runtime.actor.KryoSerializerSpec$Test1)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:557)
at com.ing.baker.runtime.actor.KryoSerializerSpec$$anonfun$1.apply(KryoSerializerSpec.scala:18)
at com.ing.baker.runtime.actor.KryoSerializerSpec$$anonfun$1.apply(KryoSerializerSpec.scala:11)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
at org.scalatest.FunSuite.withFixture(FunSuite.scala:1560)
at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183)
at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196)
at org.scalatest.FunSuite.runTest(FunSuite.scala:1560)
at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
at org.scalatest.Suite$class.run(Suite.scala:1147)
at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
at org.scalatest.FunSuite.run(FunSuite.scala:1560)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1340)
at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1334)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1334)
at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1011)
at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1010)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1500)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010)
at org.scalatest.tools.Runner$.run(Runner.scala:850)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
Caused by: java.util.ConcurrentModificationException
at java.util.Vector$Itr.checkForComodification(Vector.java:1184)
at java.util.Vector$Itr.next(Vector.java:1137)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:99)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
... 68 more
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#122 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAEJdq0kz1LF8_JGSB0unnnIbIlHyz9Iks5sTxHVgaJpZM4A6UQm>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
inner case classes seem to capture references to outer even if they are not used.
If we can find some cheap way to detect this, it would really help scalding and possibly spark.
The text was updated successfully, but these errors were encountered: