[SPARK-19882][SQL] Pivot with null as the dictinct pivot value throws NPE

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to fix two problems as below:

Use previous code path to handle null in distinct pivot values

An optimisation to this was introduced to prevent each input gets evaluated on every aggregate which seems making it slow when pivotValues are too many. It seems this tightly assumes that this distinct pivot value can't be null.

I could not find a clean and easy workaround to support this and wonder if it is worth. Please guide me if anyone knows a clean and short way to fix it in this two aggregation path.

Fix to count null

Seq(Tuple1(None), Tuple1(Some(1))).toDF("a").groupBy($"a").count().show()

Before (Spark 1.6),

+----+----+---+
|   a|null|  1|
+----+----+---+
|null|   0|  0|
|   1|   0|  1|
+----+----+---+

Before (current master), <- this is currently a regression

java.lang.NullPointerException was thrown.
java.lang.NullPointerException
  at org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst$$anonfun$4.apply(PivotFirst.scala:145)
  at org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst$$anonfun$4.apply(PivotFirst.scala:143)
  at scala.collection.immutable.List.map(List.scala:273)
  at org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst.<init>(PivotFirst.scala:143)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolvePivot$$anonfun$apply$7$$anonfun$24.apply(Analyzer.scala:509)

After,

+----+----+---+
|   a|null|  1|
+----+----+---+
|null|   1|  0|
|   1|   0|  1|
+----+----+---+

It seems we should count null given

Seq(Tuple1(None), Tuple1(Some(1))).toDF("a").groupBy($"a").count().show()

+----+-----+
|   a|count|
+----+-----+
|null|    1|
|   1|    1|
+----+-----+

How was this patch tested?

Unit tests in DataFramePivotSuite.

[SparkQA]

Test build #74269 has finished for PR 17224 at commit 0476565.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Let me update this path as soon as we decide what we want in another PR for this JIRA.

[HyukjinKwon]

I am closing this per #17226 (comment)