[SPARK-27671][SQL] Fix error when casting from a nested null in a struct #24576

viirya · 2019-05-10T10:46:17Z

What changes were proposed in this pull request?

When a null in a nested field in struct, casting from the struct throws error, currently.

scala> sql("select cast(struct(1, null) as struct<a:int,b:int>)").show
scala.MatchError: NullType (of class org.apache.spark.sql.types.NullType$)                                                                             
  at org.apache.spark.sql.catalyst.expressions.Cast.castToInt(Cast.scala:447)                                                                          
  at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:635)                                                                               
  at org.apache.spark.sql.catalyst.expressions.Cast.$anonfun$castStruct$1(Cast.scala:603)

Similarly, inline table, which casts null in nested field under the hood, also throws an error.

scala> sql("select * FROM VALUES (('a', (10, null))), (('b', (10, 50))), (('c', null)) AS tab(x, y)").show                                             
org.apache.spark.sql.AnalysisException: failed to evaluate expression named_struct('col1', 10, 'col2', NULL): NullType (of class org.apache.spark.sql.t
ypes.NullType$); line 1 pos 14                                                                                                                         
  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:47)                                                     
  at org.apache.spark.sql.catalyst.analysis.ResolveInlineTables.$anonfun$convert$6(ResolveInlineTables.scala:106)

This fixes the issue.

How was this patch tested?

Added tests.

SparkQA · 2019-05-10T13:49:22Z

Test build #105307 has finished for PR 24576 at commit 397409b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala

dongjoon-hyun · 2019-05-10T21:53:20Z

sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala

@@ -2157,4 +2157,10 @@ class DataFrameSuite extends QueryTest with SharedSQLContext {
          |*(1) Range (0, 10, step=1, splits=2)""".stripMargin))
    }
  }
+
+  test("SPARK-27671: Fix analysis exception when casting null in nested field in struct") {
+    val df = sql("select * FROM VALUES (('a', (10, null))), (('b', (10, 50))), " +


nit. select -> SELECT.

SparkQA · 2019-05-11T04:31:34Z

Test build #105328 has finished for PR 24576 at commit d38b371.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-05-11T13:10:56Z

Test build #105332 has finished for PR 24576 at commit a5c2416.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2019-05-12T00:43:09Z

cc @cloud-fan

dongjoon-hyun · 2019-05-13T16:36:39Z

Retest this please.

SparkQA · 2019-05-13T19:37:52Z

Test build #105360 has finished for PR 24576 at commit a5c2416.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM. Thank you, @viirya and @gatorsmile .
Merged to master/branch-2.4.

When a null in a nested field in struct, casting from the struct throws error, currently. ```scala scala> sql("select cast(struct(1, null) as struct<a:int,b:int>)").show scala.MatchError: NullType (of class org.apache.spark.sql.types.NullType$) at org.apache.spark.sql.catalyst.expressions.Cast.castToInt(Cast.scala:447) at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:635) at org.apache.spark.sql.catalyst.expressions.Cast.$anonfun$castStruct$1(Cast.scala:603) ``` Similarly, inline table, which casts null in nested field under the hood, also throws an error. ```scala scala> sql("select * FROM VALUES (('a', (10, null))), (('b', (10, 50))), (('c', null)) AS tab(x, y)").show org.apache.spark.sql.AnalysisException: failed to evaluate expression named_struct('col1', 10, 'col2', NULL): NullType (of class org.apache.spark.sql.t ypes.NullType$); line 1 pos 14 at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:47) at org.apache.spark.sql.catalyst.analysis.ResolveInlineTables.$anonfun$convert$6(ResolveInlineTables.scala:106) ``` This fixes the issue. Added tests. Closes #24576 from viirya/cast-null. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 8b0bdaa) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

cloud-fan · 2019-05-14T04:55:33Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

+      // For primitive types, we don't reach here because the guard of `nullSafeEval`.
+      // But for nested types like struct, we might reach here for nested null type field.
+      // We won't call the returned function actually, but returns a placeholder.
+      _ => throw new SparkException(s"should not directly cast from NullType to $to.")


when will we hit this exception?

In the case of casting a nested field in a struct or array, if we find it is a null, we don't call the cast function but set the destination as null directly. If we call this cast function, we hit this exception.

We do the thing as follow:

buildCast[InternalRow](_, row => { val newRow = new GenericInternalRow(from.fields.length) var i = 0 while (i < row.numFields) { newRow.update(i, // We don't call cast function but directly set a null, if finding a null. if (row.isNullAt(i)) null else castFuncs(i)(row.get(i, from.apply(i).dataType))) i += 1 } newRow })

Which means, it's a bug if the exception is thrown?

If reaching here, means we call the cast function for a null, unnecessarily.

When a null in a nested field in struct, casting from the struct throws error, currently. ```scala scala> sql("select cast(struct(1, null) as struct<a:int,b:int>)").show scala.MatchError: NullType (of class org.apache.spark.sql.types.NullType$) at org.apache.spark.sql.catalyst.expressions.Cast.castToInt(Cast.scala:447) at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:635) at org.apache.spark.sql.catalyst.expressions.Cast.$anonfun$castStruct$1(Cast.scala:603) ``` Similarly, inline table, which casts null in nested field under the hood, also throws an error. ```scala scala> sql("select * FROM VALUES (('a', (10, null))), (('b', (10, 50))), (('c', null)) AS tab(x, y)").show org.apache.spark.sql.AnalysisException: failed to evaluate expression named_struct('col1', 10, 'col2', NULL): NullType (of class org.apache.spark.sql.t ypes.NullType$); line 1 pos 14 at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:47) at org.apache.spark.sql.catalyst.analysis.ResolveInlineTables.$anonfun$convert$6(ResolveInlineTables.scala:106) ``` This fixes the issue. Added tests. Closes apache#24576 from viirya/cast-null. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 8b0bdaa) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

Fix error when casting from a nested null in a struct.

397409b

dongjoon-hyun reviewed May 10, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala Show resolved Hide resolved

dongjoon-hyun reviewed May 10, 2019

View reviewed changes

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala Show resolved Hide resolved

dongjoon-hyun reviewed May 10, 2019

View reviewed changes

Address comment.

d38b371

Add one more test.

a5c2416

dongjoon-hyun approved these changes May 13, 2019

View reviewed changes

dongjoon-hyun closed this in 8b0bdaa May 13, 2019

cloud-fan reviewed May 14, 2019

View reviewed changes

viirya deleted the cast-null branch December 27, 2023 18:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-27671][SQL] Fix error when casting from a nested null in a struct #24576

[SPARK-27671][SQL] Fix error when casting from a nested null in a struct #24576

viirya commented May 10, 2019 •

edited

SparkQA commented May 10, 2019

dongjoon-hyun May 10, 2019

SparkQA commented May 11, 2019

SparkQA commented May 11, 2019

gatorsmile commented May 12, 2019

dongjoon-hyun commented May 13, 2019

SparkQA commented May 13, 2019

dongjoon-hyun left a comment

cloud-fan May 14, 2019

viirya May 14, 2019

viirya May 14, 2019

cloud-fan May 14, 2019

viirya May 14, 2019

[SPARK-27671][SQL] Fix error when casting from a nested null in a struct #24576

[SPARK-27671][SQL] Fix error when casting from a nested null in a struct #24576

Conversation

viirya commented May 10, 2019 • edited

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented May 10, 2019

dongjoon-hyun May 10, 2019

Choose a reason for hiding this comment

SparkQA commented May 11, 2019

SparkQA commented May 11, 2019

gatorsmile commented May 12, 2019

dongjoon-hyun commented May 13, 2019

SparkQA commented May 13, 2019

dongjoon-hyun left a comment

Choose a reason for hiding this comment

cloud-fan May 14, 2019

Choose a reason for hiding this comment

viirya May 14, 2019

Choose a reason for hiding this comment

viirya May 14, 2019

Choose a reason for hiding this comment

cloud-fan May 14, 2019

Choose a reason for hiding this comment

viirya May 14, 2019

Choose a reason for hiding this comment

viirya commented May 10, 2019 •

edited