Commits on Apr 24, 2014

1. Register PrimitiveKeyOpenHashMap in GraphKryoRegistrator …

   The EdgePartition.index field is a PrimitiveKeyOpenHashMap[VertexId, Int],
   but this is not registered with Kryo, sometimes causing the
   exception below. This commit registers this class.
   
   com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Can not set final org.apache.spark.graphx.util.collection.PrimitiveKeyOpenHashMap field org.apache.spark.graphx.impl.EdgePartition.index to scala.collection.immutable.$colon$colon
   Serialization trace:
   index (org.apache.spark.graphx.impl.EdgePartition)
           com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:626)
           com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
           com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
           com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:43)
           com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:34)
           com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
           org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:111)
           org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:124)
           org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
           org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:27)
           scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
           scala.collection.Iterator$class.foreach(Iterator.scala:727)
           scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
           scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
           scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
           org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:100)
           org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
           org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:87)
           org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
           org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
           org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
           org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
           org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
           org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:160)
           org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:101)
           org.apache.spark.scheduler.Task.run(Task.scala:50)
           org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:212)
           org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:43)
           org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
           java.security.AccessController.doPrivileged(Native Method)
           javax.security.auth.Subject.doAs(Subject.java:415)
           org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
           org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:42)
           org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
           java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
           java.lang.Thread.run(Thread.java:744)

   

   ankurdave committed Apr 24, 2014
   Copy the full SHA

   d759a6d View commit details

   Browse the repository at this point in the history