Skip to content

Commit

Permalink
[SPARK-24640][SQL] Return NULL from size(NULL) by default
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Set the default value of the `spark.sql.legacy.sizeOfNull` config to `false`. That changes behavior of the `size()` function for `NULL`. The function will return `NULL` for `NULL` instead of `-1`.

### Why are the changes needed?
There is the agreement in the PR #21598 (comment) to change behavior in Spark 3.0.

### Does this PR introduce any user-facing change?
Yes.
Before:
```sql
spark-sql> select size(NULL);
-1
```
After:
```sql
spark-sql> select size(NULL);
NULL
```

### How was this patch tested?
By the `check outputs of expression examples` test in `SQLQuerySuite` which runs expression examples.

Closes #26051 from MaxGekk/sizeof-null-returns-null.

Authored-by: Maxim Gekk <max.gekk@gmail.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
MaxGekk authored and HyukjinKwon committed Oct 8, 2019
1 parent ef1e849 commit 4e6d31f
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
Expand Up @@ -79,7 +79,7 @@ trait BinaryArrayExpressionWithImplicitCast extends BinaryExpression
_FUNC_(expr) - Returns the size of an array or a map.
The function returns -1 if its input is null and spark.sql.legacy.sizeOfNull is set to true.
If spark.sql.legacy.sizeOfNull is set to false, the function returns null for null input.
By default, the spark.sql.legacy.sizeOfNull parameter is set to true.
By default, the spark.sql.legacy.sizeOfNull parameter is set to false.
""",
examples = """
Examples:
Expand All @@ -88,7 +88,7 @@ trait BinaryArrayExpressionWithImplicitCast extends BinaryExpression
> SELECT _FUNC_(map('a', 1, 'b', 2));
2
> SELECT _FUNC_(NULL);
-1
NULL
""")
case class Size(child: Expression) extends UnaryExpression with ExpectsInputTypes {

Expand Down
Expand Up @@ -1865,7 +1865,7 @@ object SQLConf {
.doc("If it is set to true, size of null returns -1. This behavior was inherited from Hive. " +
"The size function returns null for null input if the flag is disabled.")
.booleanConf
.createWithDefault(true)
.createWithDefault(false)

val LEGACY_REPLACE_DATABRICKS_SPARK_AVRO_ENABLED =
buildConf("spark.sql.legacy.replaceDatabricksSparkAvro.enabled")
Expand Down

0 comments on commit 4e6d31f

Please sign in to comment.