[SPARK-47646][SQL] Make try_to_number return NULL for malformed input

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to add NULL check after parsing the number so the output can be safely null for try_to_number expression.

import org.apache.spark.sql.functions._
val df = spark.createDataset(spark.sparkContext.parallelize(Seq("11")))
df.select(try_to_number($"value", lit("$99.99"))).show()

java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.types.Decimal.toPlainString()" because "<local7>" is null
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.serializefromobject_doConsume_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:50)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:894)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:894)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:368)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:332)

Why are the changes needed?

To fix the bug, and let try_to_number return NULL for malformed input as designed.

Does this PR introduce any user-facing change?

Yes, it fixes a bug. Previously, try_to_number failed with NPE.

How was this patch tested?

Unittest was added.

Was this patch authored or co-authored using generative AI tooling?

No.

[HyukjinKwon]

cc @bersprockets @gengliangwang @cloud-fan

[cloud-fan]

good catch!

[HyukjinKwon]

Merged to master and branch-3.5.

[bersprockets]

Yikes and thanks!

Will there be a 3.4.3? This also happens in 3.4.2 as well, although it takes more work to make it happen:

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.4.2
      /_/
         
Using Scala version 2.12.17 (Java HotSpot(TM) 64-Bit Server VM, Java 17.0.7)
Type in expressions to have them evaluated.
Type :help for more information.

scala> sql("create or replace temp view v1(value) as values ('11')")
res0: org.apache.spark.sql.DataFrame = []

scala> sql("cache table v1")
res1: org.apache.spark.sql.DataFrame = []

scala> val df = sql("select try_to_number(value, '$99.99') as x from v1")
df: org.apache.spark.sql.DataFrame = [x: decimal(4,2)]

scala> df.selectExpr("x + 1").show()
24/03/29 10:27:45 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 2)
java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.types.Decimal.$plus(org.apache.spark.sql.types.Decimal)" because "<local6>" is null
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:888)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:888)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)

[HyukjinKwon]

Yeah let me backport

[HyukjinKwon]

Merged to bracnh-3.4 too.

[dongjoon-hyun]

For the record, this broke branch-3.4 and the following PR fixed it.

• [SPARK-47646][SQL][FOLLOWUP][3.4] Replace non-existing try_to_number function with TryToNumber #45785

[bersprockets]

@HyukjinKwon

I should have mentioned that try_to_number exists in 3.4.2 as an SQL function but not as a scala function in functions.scala (that's why my 3.4.2 example had to use spark.sql).

[HyukjinKwon]

Thank you guys!

[gengliangwang]

Late LGTM!