Skip to content

Commit

Permalink
[SPARK-20125][SQL] Dataset of type option of map does not work
Browse files Browse the repository at this point in the history
When we build the deserializer expression for map type, we will use `StaticInvoke` to call `ArrayBasedMapData.toScalaMap`, and declare the return type as `scala.collection.immutable.Map`. If the map is inside an Option, we will wrap this `StaticInvoke` with `WrapOption`, which requires the input to be `scala.collect.Map`. Ideally this should be fine, as `scala.collection.immutable.Map` extends `scala.collect.Map`, but our `ObjectType` is too strict about this, this PR fixes it.

new regression test

Author: Wenchen Fan <wenchen@databricks.com>

Closes #17454 from cloud-fan/map.

(cherry picked from commit d4fac41)
Signed-off-by: Cheng Lian <lian@databricks.com>
  • Loading branch information
cloud-fan authored and liancheng committed Mar 28, 2017
1 parent 4bcb7d6 commit fd2e406
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 0 deletions.
Expand Up @@ -44,4 +44,9 @@ case class ObjectType(cls: Class[_]) extends DataType {
def asNullable: DataType = this

override def simpleString: String = cls.getName

override def acceptsType(other: DataType): Boolean = other match {
case ObjectType(otherCls) => cls.isAssignableFrom(otherCls)
case _ => false
}
}
Expand Up @@ -1072,10 +1072,16 @@ class DatasetSuite extends QueryTest with SharedSQLContext {
val ds2 = Seq(WithMap("hi", Map(42L -> "foo"))).toDS
checkDataset(ds2.map(t => t), WithMap("hi", Map(42L -> "foo")))
}

test("SPARK-20125: option of map") {
val ds = Seq(WithMapInOption(Some(Map(1 -> 1)))).toDS()
checkDataset(ds, WithMapInOption(Some(Map(1 -> 1))))
}
}

case class WithImmutableMap(id: String, map_test: scala.collection.immutable.Map[Long, String])
case class WithMap(id: String, map_test: scala.collection.Map[Long, String])
case class WithMapInOption(m: Option[scala.collection.Map[Int, Int]])

case class Generic[T](id: T, value: Double)

Expand Down

0 comments on commit fd2e406

Please sign in to comment.