[SPARK-12258] [SQL] Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

[gatorsmile]

This PR is for fixing the following test case:

  test("Timestamp UDF and Null value") {
    hiveContext.runSqlHive("CREATE TABLE ts_test (ts TIMESTAMP) STORED AS TEXTFILE")
    hiveContext.runSqlHive("INSERT INTO TABLE ts_test VALUES(Null)")
    hiveContext.udf.register("dummy",
      (ts: Timestamp) => ts
    )
    val result = hiveContext.sql("SELECT dummy(ts) FROM ts_test").collect().mkString("\n")
    assertResult("[null]")(result)
  }

If users use BigDecimal, Date and Timestamp in UDF, UDF returns null if the input is null.

@cloud-fan I just followed your PR #9770 and add three more types. Could you review this PR?

Thank you!

[cloud-fan]

Can you add more details to explain why Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value? It looks weird to me and is diffrent from the primitive problems I fixed in that PR.

For primitive types, there is no way to express null unless user use boxed type in UDF parameters, but Timestamp doesn't have this limit.

[SparkQA]

Test build #47522 has finished for PR 10249 at commit 0dfa110.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

The timestamp is bound to this specific number because CodeGen uses -1L as its default (null) value for Timestamp (assuming that your timezone is GMT-8)

[davies]

I think the root cause is that the UDF does not handle null value for DateType/TimestampType/DecimalType correctly, we should pass null in instead of skipping the UDF (returning null).

[gatorsmile]

Thank you for your comments! @cloud-fan @hvanhovell @davies

After digging it further, I just realized BigDecimal does not have such an issue. In the codeGenerator.java, we always treats TimestampType as LongType and treats DateType as IntegerType.

  def javaType(dt: DataType): String = dt match {
    case BooleanType => JAVA_BOOLEAN
    case ByteType => JAVA_BYTE
    case ShortType => JAVA_SHORT
    case IntegerType | DateType => JAVA_INT
    case LongType | TimestampType => JAVA_LONG
    case FloatType => JAVA_FLOAT

In all the codegen codes, we do not have separate logic for processing DateType and TimestampType. I am wondering if the current fix is ok?

Thank you again!

[yhuai]

test this please

[yhuai]

@gatorsmile Do we have other data types that still have this issue?

[gatorsmile]

Let me do more tests. Will let you know soon.

[yhuai]

test this please

[yhuai]

@gatorsmile @davies has a fix that should fix the problem fundamentally (#10259). You can take a look and try it out.

[gatorsmile]

Let me close it. Thanks!

[SparkQA]

Test build #47547 has finished for PR 10249 at commit 446d1fa.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.