[SPARK-19980][SQL][BACKPORT-2.1] Add NULL checks in Bean serializer

[maropu]

What changes were proposed in this pull request?

A Bean serializer in ExpressionEncoder could change values when Beans having NULL. A concrete example is as follows;

scala> :paste
class Outer extends Serializable {
  private var cls: Inner = _
  def setCls(c: Inner): Unit = cls = c
  def getCls(): Inner = cls
}

class Inner extends Serializable {
  private var str: String = _
  def setStr(s: String): Unit = str = str
  def getStr(): String = str
}

scala> Seq("""{"cls":null}""", """{"cls": {"str":null}}""").toDF().write.text("data")
scala> val encoder = Encoders.bean(classOf[Outer])
scala> val schema = encoder.schema
scala> val df = spark.read.schema(schema).json("data").as[Outer](encoder)
scala> df.show
+------+
|   cls|
+------+
|[null]|
|  null|
+------+

scala> df.map(x => x)(encoder).show()
+------+
|   cls|
+------+
|[null]|
|[null]|     // <-- Value changed
+------+

This is because the Bean serializer does not have the NULL-check expressions that the serializer of Scala's product types has. Actually, this value change does not happen in Scala's product types;

scala> :paste
case class Outer(cls: Inner)
case class Inner(str: String)

scala> val encoder = Encoders.product[Outer]
scala> val schema = encoder.schema
scala> val df = spark.read.schema(schema).json("data").as[Outer](encoder)
scala> df.show
+------+
|   cls|
+------+
|[null]|
|  null|
+------+

scala> df.map(x => x)(encoder).show()
+------+
|   cls|
+------+
|[null]|
|  null|
+------+

This pr added the NULL-check expressions in Bean serializer along with the serializer of Scala's product types.

How was this patch tested?

Added tests in JavaDatasetSuite.

[SparkQA]

Test build #74961 has finished for PR 17372 at commit 89305b6.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #74966 has finished for PR 17372 at commit cb7cccc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

@cloud-fan could you check this?

[cloud-fan]

thanks, merging to branch 2.1