Skip to content

Commit

Permalink
[SPARK-12834] Change ser/de of JavaArray and JavaList
Browse files Browse the repository at this point in the history
https://issues.apache.org/jira/browse/SPARK-12834

We use `SerDe.dumps()` to serialize `JavaArray` and `JavaList` in `PythonMLLibAPI`, then deserialize them with `PickleSerializer` in Python side. However, there is no need to transform them in such an inefficient way. Instead of it, we can use type conversion to convert them, e.g. `list(JavaArray)` or `list(JavaList)`. What's more, there is an issue to Ser/De Scala Array as I said in https://issues.apache.org/jira/browse/SPARK-12780

Author: Xusen Yin <yinxusen@gmail.com>

Closes apache#10772 from yinxusen/SPARK-12834.
  • Loading branch information
yinxusen authored and jkbradley committed Jan 27, 2016
1 parent 85518ed commit 4a621bc
Showing 1 changed file with 5 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -1464,7 +1464,11 @@ private[spark] object SerDe extends Serializable {
initialize()

def dumps(obj: AnyRef): Array[Byte] = {
new Pickler().dumps(obj)
obj match {
// Pickler in Python side cannot deserialize Scala Array normally. See SPARK-12834.
case array: Array[_] => new Pickler().dumps(array.toSeq.asJava)
case _ => new Pickler().dumps(obj)
}
}

def loads(bytes: Array[Byte]): AnyRef = {
Expand Down

0 comments on commit 4a621bc

Please sign in to comment.