update shortest path test to make it fail on spark 1.5 and 1.6

[mengxr]

This could be a bug in our integral ID mapping.

I can reproduce the bug in Spark 1.6:

val df = sqlContext.range(10).select(col("id"), monotonicallyIncreasingId().as("long_id”))
df.show()

+---+-----------+
| id|    long_id|
+---+-----------+
|  0|          0|
|  1|          1|
|  2| 8589934592|
|  3| 8589934593|
|  4| 8589934594|
|  5|17179869184|
|  6|17179869185|
|  7|25769803776|
|  8|25769803777|
|  9|25769803778|
+---+-----------+

df.filter(col("id") === 3).show()

+---+----------+
| id|   long_id|
+---+----------+
|  3|8589934592|
+---+----------+

If seems that filter is pushed down below monotonicallyIncreadingId, which is wrong.

== Physical Plan ==
Project [id#837L,monotonicallyincreasingid() AS long_id#838L]
+- Filter (id#837L = 3)
   +- Scan ExistingRDD[id#837L]

@rxin

[liancheng]

I'm looking into this one.

[liancheng]

Please note that due to implementation strategy of MonotonicallyIncreadingID, DataFrame partition number is crucial for reproducing this bug. We should filter out an element that belongs to any partition other than the 1st one, and this element must not be the 1st element in that partition.

For example, on my 8-core laptop, Spark default parallelism is 8, the following code reproduces this issue:

import org.apache.spark.sql.functions._

val df = sqlContext
  .range(16) // 8 partitions, 2 elements per partition
  .select(
    col("id"),
    monotonicallyIncreasingId().as("long_id")
  )

df.show()
df.filter(col("id") === 3) // 2nd element in the 2nd partition
  .show()

[mengxr]

@liancheng Could you create a Spark JIRA and post the link here? We will implement a workaround in graphframes while waiting for the official fix. Thanks!

[liancheng]

@mengxr Sorry, missed your last comment. JIRA link: https://issues.apache.org/jira/browse/SPARK-13473