[Scala] Symbol is not serializable

[lutianming]

Environment info

Operating System: debian testing

Compiler:

Package used (Python/R/Scala/Julia): Scala 2.11, Spark 2.0.1

MXNet version:

Or if installed from source: from master 2017-01-16

MXNet commit hash (git rev-parse HEAD):

Error Message:

2017-01-17 16:20:38,890 ERROR executor.Executor (Logging.scala:logError(91)) - Exception in task 0.0 in stage 1.0 (TID 1)
java.io.NotSerializableException: ml.dmlc.mxnet.Symbol
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at ml.dmlc.mxnet.JavaSerializer.serialize(Serializer.scala:48)
at ml.dmlc.mxnet.KVStore.setOptimizer(KVStore.scala:187)
at ml.dmlc.mxnet.Model$$anonfun$trainMultiDevice$4.apply(Model.scala:259)
at ml.dmlc.mxnet.Model$$anonfun$trainMultiDevice$4.apply(Model.scala:259)
at scala.Option.foreach(Option.scala:257)
at ml.dmlc.mxnet.Model$.trainMultiDevice(Model.scala:259)
at ml.dmlc.mxnet.FeedForward.fit(FeedForward.scala:347)
at ml.dmlc.mxnet.FeedForward.fit(FeedForward.scala:286)
at ml.dmlc.mxnet.FeedForward.fit(FeedForward.scala:294)
at ml.dmlc.mxnet.FeedForward.fit(FeedForward.scala:300)
at ml.dmlc.mxnet.spark.MXNet$$anonfun$1.apply(MXNet.scala:168)
at ml.dmlc.mxnet.spark.MXNet$$anonfun$1.apply(MXNet.scala:127)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:785)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:785)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:332)
at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:330)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:935)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:926)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:926)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:670)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:281)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Minimum reproducible example

Steps to reproduce

run mxnet/scala-package/spark/bin/run-mnist-example.sh example

What have you tried to solve it?

it seems that the commit [Scala] Bucketing API Support which adds Symbol to Optimizer causes the problem.

1. since symbol is not used right now, just remove it from Optimizer temporarily
2. or convert symbol to json in Optimizer

[yzhliu]

I didn't test the spark task when I built the bucketing api. Sorry for that.
Seems that we can solve it by modifying the implementation of ml.dmlc.mxnet.Serializer.

[yxzf]

@Javelinjs Did you solve it? I still have the problem with the latest source code.

[Roshrini]

@yxzf I also got the same issue. For now, if you want to run the example, you can remove symbol code from ml.dmlc.mxnet.Optimizer temporarily and it will work fine.

[qiyuangong]

@Roshrini @yxzf
You can mark symbol as transient to solve this problem ( i.e., @transient protected var symbol: Symbol = null in line 53) , which means don't Serializer/Derializer this variable. Then, everything goes on well.

Thanks to @huafengw . 👍

[yajiedesign]

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!