Running into problems when pickling Map and HashMap

[vchuravy]

steps (0.10.0)

scala-pickling> console
[info] Starting scala interpreter...
[info] 
Welcome to Scala version 2.11.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import scala.collection.immutable
import scala.collection.immutable

scala> import scala.pickling._, Defaults._, json._
import scala.pickling._
import Defaults._
import json._

scala> immutable.HashMap(( for (j <- 0 until 20) yield j -> "Test" ): _* ).pickle.value

problem (0.10.0)

scala> println(immutable.HashMap(( for (j <- 0 until 20) yield j -> "Test" ): _* ).pickle.value)
{
  "$type": "scala.collection.immutable.HashMap.HashTrieMap",
  "bitmap": 2044127639,
  "elems": {
    "elems": [
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    },
      {
      "$type": "scala.collection.immutable.HashMap"
    }
    ]
  },
  "size0": 20
}

original report

My data structure is stored as a Map and when I try to pickle it I run into some weird issues.

When pickling a immutable.HashMap using the following example code.

import scala.collection.immutable
import scala.pickling._
import json._

object Test {
  def main(args : Array[String]) {
    println(immutable.HashMap(( for (j <- 0 until 20) yield j -> "Test" ): _* ).pickle.toString)
  }
}

as output I get

JSONPickle({
  "tpe": "scala.collection.immutable.HashMap.HashTrieMap",
  "bitmap": 2044127639,
  "elems": {

  },
  "size0": 20
})

If I use instead the following snippet of code

import scala.collection.Map
import scala.pickling._
import json._

object Test {
  def main(args : Array[String]) {
    println(Map(( for (j <- 0 until 20) yield j -> "Test" ): _* ).pickle.toString)
  }
}

I also end up with

JSONPickle({
  "tpe": "scala.collection.immutable.HashMap.HashTrieMap",
  "bitmap": 2044127639,
  "elems": {

  },
  "size0": 20
})

only when I don't explicitly import Map do I end up with the correct output.

[LBliss]

Most likely due to the recurring problem with anonymous classes / traits and their current unsatisfactory fix (https://github.com/scala/pickling/blob/2.10.x/core/src/main/scala/pickling/Compat.scala).

This is currently under investigation to find a more suitable solution.

Related to #32, #38 and possible others.

[vchuravy]

Yeah a fix would be very nice. because that makes it quite hard to work with nested data. Is there a workaround?

[vchuravy]

When i do not import Map at all I run into this compile time error:

scala: 
     while compiling: /home/wallnuss/Evo2DSim/src/main/scala/org/vastness/evo2dsim/evolution/SUSEvolution.scala
        during phase: jvm
     library version: version 2.10.3
    compiler version: version 2.10.3
  reconstructed args: -bootclasspath /opt/java/jre/lib/resources.jar:/opt/java/jre/lib/rt.jar:/opt/java/jre/lib/sunrsasign.jar:/opt/java/jre/lib/jsse.jar:/opt/java/jre/lib/jce.jar:/opt/java/jre/lib/charsets.jar:/opt/java/jre/lib/jfr.jar:/opt/java/jre/classes:/home/wallnuss/.m2/repository/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.jar -classpath /opt/java/jre/lib/jsse.jar:/opt/java/jre/lib/jfxrt.jar:/opt/java/jre/lib/jfr.jar:/opt/java/jre/lib/rt.jar:/opt/java/jre/lib/resources.jar:/opt/java/jre/lib/javaws.jar:/opt/java/jre/lib/plugin.jar:/opt/java/jre/lib/management-agent.jar:/opt/java/jre/lib/jce.jar:/opt/java/jre/lib/charsets.jar:/opt/java/jre/lib/deploy.jar:/opt/java/jre/lib/ext/sunjce_provider.jar:/opt/java/jre/lib/ext/sunec.jar:/opt/java/jre/lib/ext/dnsns.jar:/opt/java/jre/lib/ext/sunpkcs11.jar:/opt/java/jre/lib/ext/localedata.jar:/opt/java/jre/lib/ext/zipfs.jar:/home/wallnuss/Evo2DSim/target/classes:/home/wallnuss/.m2/repository/org/scala-lang/scala-reflect/2.10.0/scala-reflect-2.10.0.jar:/home/wallnuss/.m2/repository/org/jbox2d/jbox2d-library/2.2.1.1/jbox2d-library-2.2.1.1.jar:/home/wallnuss/.m2/repository/com/typesafe/akka/akka-actor_2.10/2.2.1/akka-actor_2.10-2.2.1.jar:/home/wallnuss/.m2/repository/com/typesafe/config/1.0.2/config-1.0.2.jar:/home/wallnuss/.m2/repository/com/intellij/forms_rt/7.0.3/forms_rt-7.0.3.jar:/home/wallnuss/.m2/repository/asm/asm-commons/3.0/asm-commons-3.0.jar:/home/wallnuss/.m2/repository/asm/asm-tree/3.0/asm-tree-3.0.jar:/home/wallnuss/.m2/repository/asm/asm/3.0/asm-3.0.jar:/home/wallnuss/.m2/repository/com/jgoodies/forms/1.1-preview/forms-1.1-preview.jar:/home/wallnuss/.m2/repository/jdom/jdom/1.0/jdom-1.0.jar:/home/wallnuss/.m2/repository/com/github/scopt/scopt_2.10/3.1.0/scopt_2.10-3.1.0.jar:/home/wallnuss/.m2/repository/org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar:/home/wallnuss/.m2/repository/com/github/scala-incubator/io/scala-io-file_2.10/0.4.2/scala-io-file_2.10-0.4.2.jar:/home/wallnuss/.m2/repository/com/github/scala-incubator/io/scala-io-core_2.10/0.4.2/scala-io-core_2.10-0.4.2.jar:/home/wallnuss/.m2/repository/com/jsuereth/scala-arm_2.10/1.3/scala-arm_2.10-1.3.jar:/home/wallnuss/.m2/repository/org/scala-lang/scala-pickling_2.10/0.8.0-SNAPSHOT/scala-pickling_2.10-0.8.0-SNAPSHOT.jar
  last tree to typer: Literal(Constant(anon$93))
              symbol: null
   symbol definition: null
                 tpe: Class(classOf[org.vastness.evo2dsim.evolution.Evolution$$anon$93])
       symbol owners: 
      context owners: anonymous class anonfun$normalizeResults$2 -> package evolution
== Enclosing template or block ==
Template( // val <local $anonfun>: <notype>, tree.tpe=org.vastness.evo2dsim.evolution.anonfun$normalizeResults$2
  "scala.runtime.AbstractFunction1", "scala.Serializable" // parents
  ValDef(
    private
    "_"
    <tpt>
    <empty>
  )
  // 6 statements
  DefDef( // final def apply(x$4: Tuple2): Tuple2
    <method> final <triedcooking>
    "apply"
    []
    // 1 parameter list
    ValDef( // x$4: Tuple2
      <param> <synthetic> <triedcooking>
      "x$4"
      <tpt> // tree.tpe=Tuple2
      <empty>
    )
    <tpt> // tree.tpe=Tuple2
    Block( // tree.tpe=Tuple2
      // 3 statements
      ValDef( // case val x1: Tuple2
        case <synthetic> <triedcooking>
        "x1"
        <tpt> // tree.tpe=Tuple2
        Typed( // tree.tpe=Tuple2
          "x$4" // x$4: Tuple2, tree.tpe=Tuple2
          <tpt> // tree.tpe=Tuple2
        )
      )
      LabelDef( // case def case6(): Tuple2, tree.tpe=Tuple2
        ()
        If( // tree.tpe=Tuple2
          Apply( // final def ne(x$1: Object): Boolean in class Object, tree.tpe=Boolean
            "x1"."ne" // final def ne(x$1: Object): Boolean in class Object, tree.tpe=(x$1: Object)Boolean
            null
          )
          Block( // tree.tpe=Tuple2
            // 2 statements
            ValDef( // val key: Int
              <triedcooking>
              "key"
              <tpt> // tree.tpe=Int
              Apply( // val _1$mcI$sp(): Int in class Tuple2, tree.tpe=Int
                "x1"."_1$mcI$sp" // val _1$mcI$sp(): Int in class Tuple2, tree.tpe=()Int
                Nil
              )
            )
            ValDef( // val p2: Tuple2
              <synthetic> <triedcooking>
              "p2"
              <tpt> // tree.tpe=Tuple2
              Apply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=Tuple2
                TypeApply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=()Tuple2
                  x1._2()."$asInstanceOf" // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=[T0 >: ? <: ?]()T0
                  <tpt> // tree.tpe=Tuple2
                )
                Nil
              )
            )
            If( // tree.tpe=Tuple2
              Apply( // final def ne(x$1: Object): Boolean in class Object, tree.tpe=Boolean
                "p2"."ne" // final def ne(x$1: Object): Boolean in class Object, tree.tpe=(x$1: Object)Boolean
                null
              )
              Block( // tree.tpe=Tuple2
                ValDef( // val fitness: Double
                  <triedcooking>
                  "fitness"
                  <tpt> // tree.tpe=Double
                  Apply( // val _1$mcD$sp(): Double in class Tuple2, tree.tpe=Double
                    "p2"."_1$mcD$sp" // val _1$mcD$sp(): Double in class Tuple2, tree.tpe=()Double
                    Nil
                  )
                )
                Apply( // case def matchEnd5(x: Tuple2): Tuple2, tree.tpe=Tuple2
                  "matchEnd5" // case def matchEnd5(x: Tuple2): Tuple2, tree.tpe=(x: Tuple2)Tuple2
                  Apply( // final def ->$extension($this: Object,y: Object): Tuple2 in object Predef$ArrowAssoc, tree.tpe=Tuple2
                    Predef$ArrowAssoc.this."$minus$greater$extension" // final def ->$extension($this: Object,y: Object): Tuple2 in object Predef$ArrowAssoc, tree.tpe=($this: Object, y: Object)Tuple2
                    // 2 arguments
                    Apply( // implicit def any2ArrowAssoc(x: Object): Object in object Predef, tree.tpe=Object
                      scala.this."Predef"."any2ArrowAssoc" // implicit def any2ArrowAssoc(x: Object): Object in object Predef, tree.tpe=(x: Object)Object
                      Apply( // def box(x: Int): Integer in object Int, tree.tpe=Object
                        "scala"."Int"."box" // def box(x: Int): Integer in object Int, tree.tpe=(x: Int)Integer
                        "key" // val key: Int, tree.tpe=Int
                      )
                    )
                    Apply( // def box(x: Double): Double in object Double, tree.tpe=Object
                      "scala"."Double"."box" // def box(x: Double): Double in object Double, tree.tpe=(x: Double)Double
                      Apply( // final def f_norm$1(x: Double,abs_rel$1: Function1,total$1: Double): Double in class SUSEvolution, tree.tpe=Double
                        SUSEvolution$$anonfun$normalizeResults$2.this."$outer "."org$vastness$evo2dsim$evolution$SUSEvolution$$f_norm$1" // final def f_norm$1(x: Double,abs_rel$1: Function1,total$1: Double): Double in class SUSEvolution, tree.tpe=(x: Double, abs_rel$1: Function1, total$1: Double)Double
                        // 3 arguments
                        "fitness" // val fitness: Double, tree.tpe=Double
                        SUSEvolution$$anonfun$normalizeResults$2.this."abs_rel$1" // private[this] val abs_rel$1: Function1, tree.tpe=Function1
                        SUSEvolution$$anonfun$normalizeResults$2.this."total$1" // private[this] val total$1: Double, tree.tpe=Double
                      )
                    )
                  )
                )
              )
              Apply( // case def case7(): Tuple2, tree.tpe=Tuple2
                "case7" // case def case7(): Tuple2, tree.tpe=()Tuple2
                Nil
              )
            )
          )
          Apply( // case def case7(): Tuple2, tree.tpe=Tuple2
            "case7" // case def case7(): Tuple2, tree.tpe=()Tuple2
            Nil
          )
        )
      )
      LabelDef( // case def case7(): Tuple2, tree.tpe=Tuple2
        ()
        Apply( // case def matchEnd5(x: Tuple2): Tuple2, tree.tpe=Tuple2
          "matchEnd5" // case def matchEnd5(x: Tuple2): Tuple2, tree.tpe=(x: Tuple2)Tuple2
          Throw( // tree.tpe=Nothing
            Apply( // def <init>(obj: Object): MatchError in class MatchError, tree.tpe=MatchError
              new MatchError."<init>" // def <init>(obj: Object): MatchError in class MatchError, tree.tpe=(obj: Object)MatchError
              "x1" // case val x1: Tuple2, tree.tpe=Tuple2
            )
          )
        )
      )
      LabelDef( // case def matchEnd5(x: Tuple2): Tuple2, tree.tpe=Tuple2
        "x" // x: Tuple2, tree.tpe=Tuple2
        "x" // x: Tuple2, tree.tpe=Tuple2
      )
    )
  )
  ValDef( // private[this] val $outer: org.vastness.evo2dsim.evolution.SUSEvolution
    private <local> <synthetic> <paramaccessor> <triedcooking>
    "$outer "
    <tpt> // tree.tpe=org.vastness.evo2dsim.evolution.SUSEvolution
    <empty>
  )
  DefDef( // final def apply(v1: Object): Object
    <method> final <bridge>
    "apply"
    []
    // 1 parameter list
    ValDef( // v1: Object
      <param> <triedcooking>
      "v1"
      <tpt> // tree.tpe=Object
      <empty>
    )
    <tpt> // tree.tpe=Object
    Apply( // final def apply(x$4: Tuple2): Tuple2, tree.tpe=Tuple2
      SUSEvolution$$anonfun$normalizeResults$2.this."apply" // final def apply(x$4: Tuple2): Tuple2, tree.tpe=(x$4: Tuple2)Tuple2
      Apply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=Tuple2
        TypeApply( // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=()Tuple2
          "v1"."$asInstanceOf" // final def $asInstanceOf[T0 >: ? <: ?](): T0 in class Object, tree.tpe=[T0 >: ? <: ?]()T0
          <tpt> // tree.tpe=Tuple2
        )
        Nil
      )
    )
  )
  ValDef( // private[this] val abs_rel$1: Function1
    private <local> <synthetic> <paramaccessor> <triedcooking>
    "abs_rel$1"
    <tpt> // tree.tpe=Function1
    <empty>
  )
  ValDef( // private[this] val total$1: Double
    private <local> <synthetic> <paramaccessor> <triedcooking>
    "total$1"
    <tpt> // tree.tpe=Double
    <empty>
  )
  DefDef( // def <init>(arg$outer: org.vastness.evo2dsim.evolution.SUSEvolution,abs_rel$1: Function1,total$1: Double): org.vastness.evo2dsim.evolution.anonfun$normalizeResults$2
    <method> <triedcooking>
    "<init>"
    []
    // 1 parameter list
    ValDef( // $outer: org.vastness.evo2dsim.evolution.SUSEvolution
      <param> <triedcooking>
      "$outer"
      <tpt> // tree.tpe=org.vastness.evo2dsim.evolution.SUSEvolution
      <empty>
    )
    ValDef( // abs_rel$1: Function1
      <param> <synthetic> <triedcooking>
      "abs_rel$1"
      <tpt> // tree.tpe=Function1
      <empty>
    )
    ValDef( // total$1: Double
      <param> <synthetic> <triedcooking>
      "total$1"
      <tpt> // tree.tpe=Double
      <empty>
    )
    <tpt> // tree.tpe=org.vastness.evo2dsim.evolution.anonfun$normalizeResults$2
    Block( // tree.tpe=Unit
      // 4 statements
      If( // tree.tpe=Unit
        Apply( // final def eq(x$1: Object): Boolean in class Object, tree.tpe=Boolean
          "$outer"."eq" // final def eq(x$1: Object): Boolean in class Object, tree.tpe=(x$1: Object)Boolean
          null
        )
        Throw( // tree.tpe=Nothing
          Apply( // def <init>(): NullPointerException in class NullPointerException, tree.tpe=NullPointerException
            new NullPointerException."<init>" // def <init>(): NullPointerException in class NullPointerException, tree.tpe=()NullPointerException
            Nil
          )
        )
        Assign( // tree.tpe=Unit
          SUSEvolution$$anonfun$normalizeResults$2.this."$outer " // private[this] val $outer: org.vastness.evo2dsim.evolution.SUSEvolution, tree.tpe=org.vastness.evo2dsim.evolution.SUSEvolution
          "$outer" // $outer: org.vastness.evo2dsim.evolution.SUSEvolution, tree.tpe=org.vastness.evo2dsim.evolution.SUSEvolution
        )
      )
      Assign( // tree.tpe=Unit
        SUSEvolution$$anonfun$normalizeResults$2.this."abs_rel$1" // private[this] val abs_rel$1: Function1, tree.tpe=Function1
        "abs_rel$1" // abs_rel$1: Function1, tree.tpe=Function1
      )
      Assign( // tree.tpe=Unit
        SUSEvolution$$anonfun$normalizeResults$2.this."total$1" // private[this] val total$1: Double, tree.tpe=Double
        "total$1" // total$1: Double, tree.tpe=Double
      )
      Apply( // def <init>(): scala.runtime.AbstractFunction1 in class AbstractFunction1, tree.tpe=scala.runtime.AbstractFunction1
        SUSEvolution$$anonfun$normalizeResults$2.super."<init>" // def <init>(): scala.runtime.AbstractFunction1 in class AbstractFunction1, tree.tpe=()scala.runtime.AbstractFunction1
        Nil
      )
      ()
    )
  )
)
== Expanded type of tree ==
ConstantType(value = Constant(anon$93))
uncaught exception during compilation: java.io.IOException

[yigit]

this does not work either:

import scala.pickling._
import json._
("test", Map(1->(2,3))).pickle

returns

JSONPickle({
  "tpe": "scala.Tuple2[java.lang.String,scala.collection.immutable.Map[scala.Int,scala.Tuple2[scala.Int,scala.Int]]]",
  "_1": "test",
  "_2": {
    "tpe": "scala.collection.immutable.Map[scala.Int,scala.Tuple2[scala.Int,scala.Int]]",
    "elems": [
      {
      "tpe": "scala.Tuple2[scala.Int,scala.Tuple2[scala.Int,scala.Int]]",
      "_1": 1,
      "_2": {
        "tpe": "scala.Tuple2$mcII$sp",
        "_1": null,
        "_2": null
      }
    }
    ]
  }
})

the values in the tuple are both null instead of 2 & 3

[eed3si9n]

I was able to reproduce this problem using Pickling 0.10.0.

[phaller]

Looks like this issue is due to the fact that specialized classes are not
handled correctly.
On Feb 8, 2015 7:07 AM, "eugene yokota" notifications@github.com wrote:

I was able to reproduce this problem using Pickling 0.10.0.

—
Reply to this email directly or view it on GitHub
#63 (comment).

[reid-spencer]

@phaller - Looking at this example @yigit provided, It is curious that the scala.Tuple2 at the top level seems to work fine but the Tuple2 as the value of the Map does not. Just thought I'd mention it in case that's a clue to fixing this.

[jsuereth]

Just some quick notes:

• I was able to correct the reported issue by providing a new hand-rolled ImmutableMap pickler which uses TravPickler generator.
• We still have the runtime issue, because the runtime pickler kicks in for unpickle[Any] and this tries to look into class structure, which fails.
• We need some additional runtime hooks to provide custom runtime picklers but NOT for concrete types, but rather type constructors (like Map[,]). We already have a mechanism in place specifically to handle Tuple2, but it's not extensible.