Exception in thread "main" org.apache.spark.SparkException: Task not serializable

[hyunjun]

Hi, I could run the example like below;

val m1 = Map(...)
val m2 = Map(...)
spark.makeRDD(Seq(m1, m2)).saveToEs(...)

However, I couldn't convert my method to Seq

def myMethod(s: String): Map[String, String] = { ... }

def main(args: Array[String]) {
  ...
  val mine = prev.map(myMethod) // I'd like to put this result into makeRDD
  spark.makeRDD(???).saveToEs(...)
}

I did various ways, just got several errors including Task not serializable
Let me know, please. Thanks.

[costin]

It's best to ask these questions on the Spark mailing list not on the bug tracker. You'll likely find more information there.

I suggest looking at the Scala/Spark documentation - you should use closures/methods to pass it the calculation as a function not as a value that gets evaluated on the RDD at runtime. As for your error, your Task needs implement Serializable interface- in fact, if you google for it, you'll find tons on resources on the topic.

Hope this helps,