We at Spark ran into a problem using Chill to serialize WrappedArray. See the following code. It looks like the problem is with a WrappedArray for specialized tuples. If I replace the ints with string (and thus no specialized Tuple2), the exception goes away.
This is becoming a major problem for the Spark project because anything with primitive WrappedArray would fail.
scala> val ser = new spark.KryoSerializer
ser: spark.KryoSerializer = spark.KryoSerializer@7798ce82
scala> ser.newInstance.deserialize[Array[(Int, Int)]]( ser.newInstance.serialize(Array((1, 2), (2, 3))) )
res12: Array[(Int, Int)] = Array((1,2), (2,3))
scala> ser.newInstance.deserialize[Seq[(Int, String)]]( ser.newInstance.serialize(Array((1, "2"), (2, "3")).toSeq) )
res20: Seq[(Int, String)] = WrappedArray((1,2), (2,3))
scala> ser.newInstance.deserialize[Seq[(Int, Int)]]( ser.newInstance.serialize(Array((1, 2), (2, 3)).toSeq) )
java.lang.ArrayIndexOutOfBoundsException: -2
at java.util.ArrayList.get(ArrayList.java:324)
at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:773)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:624)
at com.twitter.chill.WrappedArraySerializer$$anonfun1.apply(WrappedArraySerializer.scala:44)
at com.twitter.chill.WrappedArraySerializer$$anonfun1.apply(WrappedArraySerializer.scala:43)
at scala.collection.immutable.Range.foreach(Range.scala:81)
at com.twitter.chill.WrappedArraySerializer.read(WrappedArraySerializer.scala:43)
at com.twitter.chill.WrappedArraySerializer.read(WrappedArraySerializer.scala:21)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
at spark.KryoSerializerInstance.deserialize(KryoSerializer.scala:75)
at <init>(<console>:13)
at <init>(<console>:18)
at <init>(<console>:20)
at <init>(<console>:22)
at <init>(<console>:24)
at .<init>(<console>:28)
at .<clinit>(<console>)
at .<init>(<console>:11)
at .<clinit>(<console>)
at $export(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:629)
at spark.repl.SparkIMain$Request$$anonfun$10.apply(SparkIMain.scala:890)
at scala.tools.nsc.interpreter.Line$$anonfun$1.apply$mcV$sp(Line.scala:43)
at scala.tools.nsc.io.package$$anon$2.run(package.scala:25)
at java.lang.Thread.run(Thread.java:680)
Hi guys,
We at Spark ran into a problem using Chill to serialize WrappedArray. See the following code. It looks like the problem is with a WrappedArray for specialized tuples. If I replace the ints with string (and thus no specialized Tuple2), the exception goes away.
This is becoming a major problem for the Spark project because anything with primitive WrappedArray would fail.
@mateiz, @dlyubimov