[SPARK-14063][SQL] SQLContext.range should return Dataset[java.lang.Long]

[rxin]

What changes were proposed in this pull request?

This patch changed the return type for SQLContext.range from Dataset[Long] (Scala primitive) to Dataset[java.lang.Long] (Java boxed long).

Previously, SPARK-13894 changed the return type of range from Dataset[Row] to Dataset[Long]. The problem is that due to https://issues.scala-lang.org/browse/SI-4388, Scala compiles primitive types in generics into just Object, i.e. range at bytecode level now just returns Dataset[Object]. This is really bad for Java users because they are losing type safety and also need to add a type cast every time they use range.

Talked to Jason Zaugg from Lightbend (Typesafe) who suggested the best approach is to return Dataset[java.lang.Long]. The downside is that when Scala users want to explicitly type a closure used on the dataset returned by range, they would need to use java.lang.Long instead of the Scala Long.

How was this patch tested?

The signature change should be covered by existing unit tests and API tests. I also added a new test case in DatasetSuite for range.

[cloud-fan]

LGTM

[cloud-fan]

Why does this code pass compile? Is it because we only turn scalac warnings to errors, but not javac warning?

[rxin]

Yes

[SparkQA]

Test build #53754 has finished for PR 11880 at commit 8272a1f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Good catch of course. I grepped around for similar issues and noticed this in JavaDoubleRDD:

  def histogram(bucketCount: Int): (Array[scala.Double], Array[Long]) = {
    val result = srdd.histogram(bucketCount)
    (result._1, result._2)
  }

  def histogram(buckets: Array[scala.Double]): Array[Long] = {
    srdd.histogram(buckets, false)
  }

  def histogram(buckets: Array[JDouble], evenBuckets: Boolean): Array[Long] = {
    srdd.histogram(buckets.map(_.toDouble), evenBuckets)
  }

For a second I thought it was the same problem, but I suppose an Array[scala.Long] is actually long[] and so is already fine in Java. Different from the example you're fixing.

[rxin]

@srowen yea I think arrays are fine since those are not generics.

Merging in master.