Skip to content

Commit

Permalink
[SPARK-18717][SQL] Make code generation for Scala Map work with immut…
Browse files Browse the repository at this point in the history
…able.Map also

## What changes were proposed in this pull request?

Fixes compile errors in generated code when user has case class with a `scala.collections.immutable.Map` instead of a `scala.collections.Map`. Since ArrayBasedMapData.toScalaMap returns the immutable version we can make it work with both.

## How was this patch tested?

Additional unit tests.

Author: Andrew Ray <ray.andrew@gmail.com>

Closes #16161 from aray/fix-map-codegen.
  • Loading branch information
aray authored and cloud-fan committed Dec 13, 2016
1 parent 2aa16d0 commit 46d30ac
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 1 deletion.
Expand Up @@ -342,7 +342,7 @@ object ScalaReflection extends ScalaReflection {

StaticInvoke(
ArrayBasedMapData.getClass,
ObjectType(classOf[Map[_, _]]),
ObjectType(classOf[scala.collection.immutable.Map[_, _]]),
"toScalaMap",
keyData :: valueData :: Nil)

Expand Down
12 changes: 12 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
Expand Up @@ -1120,8 +1120,20 @@ class DatasetSuite extends QueryTest with SharedSQLContext {
// sizeInBytes is 2404280404, before the fix, it overflows to a negative number
assert(sizeInBytes > 0)
}

test("SPARK-18717: code generation works for both scala.collection.Map" +
" and scala.collection.imutable.Map") {
val ds = Seq(WithImmutableMap("hi", Map(42L -> "foo"))).toDS
checkDataset(ds.map(t => t), WithImmutableMap("hi", Map(42L -> "foo")))

val ds2 = Seq(WithMap("hi", Map(42L -> "foo"))).toDS
checkDataset(ds2.map(t => t), WithMap("hi", Map(42L -> "foo")))
}
}

case class WithImmutableMap(id: String, map_test: scala.collection.immutable.Map[Long, String])
case class WithMap(id: String, map_test: scala.collection.Map[Long, String])

case class Generic[T](id: T, value: Double)

case class OtherTuple(_1: String, _2: Int)
Expand Down

0 comments on commit 46d30ac

Please sign in to comment.