#17801 Improve nullability reporting of case expressions #17813

pepijnve · 2025-09-28T15:02:38Z

Which issue does this PR close?

Closes Regression: sql_planner benchmark panic'ing on main #17801

Rationale for this change

#17357 introduced a change that replaces coalesce function calls with case expressions. In the current implementation these two differ in the way they report their nullability. coalesce is more precise than case all will report itself as not nullable in situations where the equivalent case does report being nullable.

The rest of the codebase currently does not expect the nullability property of an expression to change as a side effect of expression simplification. This PR is a first attempt to align the nullability of coalesce and case.

What changes are included in this PR?

Tweaks to the nullable logic for the logical and physical case expression code to report case as being not nullable in more situations.

Are these changes tested?

Additional unit tests have been added to test the new logic.

Are there any user-facing changes?

No

alamb

Thank you for this PR @pepijnve --

I am not quite sure about this implementation (I am hoping #17628 might solve the problem too with more sophisticated case folding)

However, I verified it does solve the problem with running the benchmarks so from that perspective I think we should proceed

My only real concern is that the newly added tests cover only the new code, and not the "end to end" behavior you tracked down (namely that the case pattern with coalesce changes the nullability).

Would it be possible to add some of the cases as expr simplification tests too? Somewhere like here?

datafusion/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

Line 3881 in 247450d

#[test]

alamb · 2025-09-29T14:22:35Z

datafusion/expr/src/expr_schema.rs

+            when(binary_expr(col("foo"), Operator::Eq, lit(5)), col("foo"))
+                .otherwise(lit(0))?,


Minor: You can probably make this more concise using the eq method, something like this:

Suggested change

when(binary_expr(col("foo"), Operator::Eq, lit(5)), col("foo"))

.otherwise(lit(0))?,

when(col("foo").eq(lit(5))), col("foo")).otherwise(lit(0))?,

likewise there is Expr::and for ands that could be used as well below

However, the current setup of using and as a prefix is pretty clear too, so maybe what you have here is actually more readable.

Ah I missed that. I was looking for prefix versions, and hadn't realised infix ones existed too.

I ended up sticking with prefix notation for the boolean combinators and infix for the rest. Using infix for the boolean made it hard to read. I've also added the SQL equivalent as a comment.

alamb · 2025-09-29T14:28:43Z

datafusion/expr/src/expr_schema.rs

        assert!(expr.nullable(&get_schema(false)).unwrap());
    }

+    fn check_nullability(


I found this a little confusing at first, because it makes an explicit assumption that expr's will never introduce nulls (in order for !expr.nullable(&get_schema(false))?, to be true). So for example, it wouldn't do the right thing with the NULLIF function NULLIF(foo, 25) or something

Maybe some comments would help

Suggested change

fn check_nullability(

/// Verifies that `expr` has `nullable` nullability when the 'foo' column is

/// null.

/// Also assumes and verifies that `expr` is NOT nullable when 'foo' is NOT null

fn check_nullability(

I've reworked the logical plan test cases already to (hopefully) make it more obvious what's going on. I hadn't given this function much thought since it was only a test thing.

alamb · 2025-09-29T14:31:19Z

datafusion/expr/src/expr_schema.rs

+        check_nullability(
+            when(binary_expr(col("foo"), Operator::Eq, lit(5)), col("foo"))
+                .otherwise(lit(0))?,
+            true,


technically this could also be reported as false, given that if foo is null, then the expr resolves to 0 (non null)

> create table t(foo int) as values (0), (NULL), (5); 0 row(s) fetched. Elapsed 0.001 seconds. > select foo, CASE WHEN foo=5 THEN foo ELSE 0 END from t; +------+---------------------------------------------------------+ | foo | CASE WHEN t.foo = Int64(5) THEN t.foo ELSE Int64(0) END | +------+---------------------------------------------------------+ | 0 | 0 | | NULL | 0 | | 5 | 5 | +------+---------------------------------------------------------+ 3 row(s) fetched. Elapsed 0.002 seconds.

However, maybe we can improve that in a follow on PR

Agreed, the const evaluation is far from complete. I tried to do something good enough for the coalesce simplification initially.
I was wondering the whole time if there isn't some existing null analysis logic somewhere in the codebase we could reuse. The best I could come up with is rewriting the full expression by replacing the then expression with literal NULL and then attempting const evaluation. But that got me worrying about planning overhead again.

alamb · 2025-09-29T14:32:17Z

datafusion/expr/src/expr_schema.rs

+            when(
+                or(
+                    is_not_null(col("foo")),
+                    binary_expr(col("foo"), Operator::Eq, lit(5)),


as above, I don't think this expression can everr be true so this overall expression is still non nullable

alamb · 2025-09-29T14:34:33Z

datafusion/expr/src/expr_schema.rs

+                col("foo"),
+            )
+            .otherwise(lit(0))?,
+            true,


Here too -- this expression is not nullabile

> select foo, CASE WHEN foo=5 OR foo IS NOT NULL THEN foo ELSE 0 END from t; +------+------------------------------------------------------------------------------+ | foo | CASE WHEN t.foo = Int64(5) OR t.foo IS NOT NULL THEN t.foo ELSE Int64(0) END | +------+------------------------------------------------------------------------------+ | 0 | 0 | | NULL | 0 | | 5 | 5 | +------+------------------------------------------------------------------------------+ 3 row(s) fetched. Elapsed 0.002 seconds.

FWIW, if you comment out the filter step (i.e. revert to the pre-patch version) all of these cases are reported as being nullable. The scope of this PR is to get at least some cases that are definitely not nullable reported as such, not ensure all cases are reported correctly.

alamb · 2025-09-29T14:35:54Z

datafusion/expr/src/expr_schema.rs

+            .otherwise(lit(0))?,
+            true,
+            get_schema,
+        )?;


Can you also please add a check with is_null in the OR clause (which should be null)

Something like the equivalent to

> select foo, CASE WHEN foo=5 OR foo IS NULL THEN foo ELSE 0 END from t; +------+--------------------------------------------------------------------------+ | foo | CASE WHEN t.foo = Int64(5) OR t.foo IS NULL THEN t.foo ELSE Int64(0) END | +------+--------------------------------------------------------------------------+ | 0 | 0 | | NULL | NULL | | 5 | 5 | +------+--------------------------------------------------------------------------+ 3 row(s) fetched. Elapsed 0.000 seconds.

Like

check_nullability( when( or( binary_expr(col("foo"), Operator::Eq, lit(5)), is_null(col("foo")), ), col("foo"), ) .otherwise(lit(0))?, true, get_schema, )?;

I've added this test case

pepijnve · 2025-09-29T16:43:07Z

I am not quite sure about this implementation (I am hoping #17628 might solve the problem too with more sophisticated case folding)

I warned you it wasn't very elegant. 😄 I don't think #17628 covers the same thing though. What we're trying to do here is get ExprSchemable::nullable to report an accurate value outside of the optimiser. Ideally you would want that to work both before and after case folding.
I agree that there is a lot of overlap in the required logic though. If you were to take the case expression, replace the then expression trees with literal NULL everywhere they occur, then perform case folding, and then perform null analysis then you would get the same result.

My only real concern is that the newly added tests cover only the new code, and not the "end to end" behavior you tracked down (namely that the case pattern with coalesce changes the nullability).

Would it be possible to add some of the cases as expr simplification tests too? Somewhere like here?

I'm not sure what kind of test you have in mind. The end to end case is (admittedly very indirectly) covered by TPC-DS query 75 and the removal of the double optimisation. If you revert the production code change in this PR, but keep the test change you'll see that it fails.

For the simplifier itself, I was wondering if there shouldn't be some internal assertions that verifies that the result of calling ScalarUDF::simplify doesn't change key aspects of the expression that the schema also encodes. Both the data type and the nullability should not change since that causes the expression and the schema to be mismatched.

pepijnve · 2025-09-30T08:44:12Z

@alamb thinking about this a bit more. I'm going to struggle expressing myself sufficiently clearly here, but I'll try to explain the idea behind what I'm doing. Maybe that can help us figure out a better way to express the idea.

What I'm trying to do is improve the accuracy of the predicate is_nullable(expr) specifically for CASE expressions.
In the current code this predicate is implemented by checking (in pseudo code) case_expr.when_then.any?((when, then) -> is_nullable(then)) || is_nullable(case_expr.else). This results in quite a few CASE expressions being reported as nullable even though they're not.

In particular there's one interesting case (pun not intended) which results from the coalesce simplification and that is CASE WHEN x IS NOT NULL THEN x ELSE y. If the expression x is nullable, is_nullable will currently report the entire case expression as nullable. The implementation does not take into account that there is a guard clause preventing the null value of x from being returned.

What I attempted to do in this PR is to look at the more general form WHEN f(x) THEN x where f(x) is some arbitrary predicate that may or may not depend on the value of x. What the code is trying to do is to const evaluate f(NULL). If it can with 100% certainty (i.e. Some is returned) and the evaluated value is false, then predicate f guarantees this particular branch of the case expression from ever returning a NULL and we can ignore the nullability of x.

I tried to implement this in a cheap, but imprecise way. My rationale was that even though it's not perfect, it's an improvement in accuracy over the current code.
A possible alternative would be to rewrite the expression corresponding to f using the binding x = null and then attempting to const evaluate using the existing const evaluation code. That introduces a dependency from the logical expression module to the optimiser though and would probably have a longer run time than the current crude approximation.

pepijnve · 2025-09-30T14:35:02Z

I've massaged the logical plan version of the code a bit further already to hopefully clarify what it's doing. I then ran the test cases with logging output rather than assertions before and after the extra filtering to illustrate what's being changed. After the change all tests pass. Before the patch it reports the following

CASE WHEN x IS NOT NULL THEN x ELSE Int32(0) END nullable? should be false, but was true
CASE WHEN NOT x IS NULL THEN x ELSE Int32(0) END nullable? should be false, but was true
CASE WHEN x IS NOT NULL AND x = Int32(5) THEN x ELSE Int32(0) END nullable? should be false, but was true
CASE WHEN x = Int32(5) AND x IS NOT NULL THEN x ELSE Int32(0) END nullable? should be false, but was true
CASE WHEN x = Int32(5) AND x IS NOT NULL OR x = bar AND x IS NOT NULL THEN x ELSE Int32(0) END nullable? should be false, but was true

pepijnve · 2025-09-30T18:30:53Z

@alamb I've taken the logical expression portion of the PR another step further which ensures correct answers for the expressions you mentioned earlier. I can complete the physical expression portion as well if you like. Unless you tell me this path is a dead end.

alamb · 2025-09-30T18:54:27Z

@alamb I've taken the logical expression portion of the PR another step further which ensures correct answers for the expressions you mentioned earlier. I can complete the physical expression portion as well if you like. Unless you tell me this path is a dead end.

Thank you -- I will try and get to this one asap. Somehow every time i think I am getting the queue of reviews under control there are like 50 new notifications ! It is a good problem to have.

pepijnve · 2025-09-30T19:18:06Z

Thank you -- I will try and get to this one asap. Somehow every time i think I am getting the queue of reviews under control there are like 50 new notifications ! It is a good problem to have.

No pressure from my side. I just write up my notes and move on to the next thing. Async delayed response is fine.

…ates

pepijnve · 2025-10-01T14:20:47Z

I experimented a bit with the rewrite + const eval approach on the physical expression side of things. While attractive and simple to implement, the downside is that it's going to be very hard to ensure the logical and physical side agree. Logical needs to work without ExecutionProps so it has less information available to it compared to the PhysicalExpr tree. I don't see a way to resolve that. As a consequence I ended up with an limited ad hoc version of const evaluation on the logical side and would have to do the same for physical which isn't really ideal from a DRY perspective.

github-actions bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate labels Sep 28, 2025

pepijnve force-pushed the issue_17801 branch from d2e613a to 88a911b Compare September 29, 2025 06:02

pepijnve mentioned this pull request Sep 29, 2025

Temporarily disable failing sql_planner benchmark query #17809

Merged

pepijnve force-pushed the issue_17801 branch from 88a911b to 482d0be Compare September 29, 2025 10:51

pepijnve marked this pull request as ready for review September 29, 2025 10:52

pepijnve force-pushed the issue_17801 branch 3 times, most recently from 4bbaa82 to 7f8d7cf Compare September 29, 2025 12:56

alamb reviewed Sep 29, 2025

View reviewed changes

pepijnve force-pushed the issue_17801 branch from 9516262 to a6ab83a Compare September 29, 2025 17:19

alamb mentioned this pull request Sep 29, 2025

Revert "Disable failing benchmark query (#17809)" #17833

Draft

pepijnve added 5 commits October 1, 2025 09:35

apache#17801 Improve nullability reporting of case expressions

0102f4b

Clarify logical expression test cases

753715e

WIP

c9190d6

apache#17838 Attempt to clarify const evaluation logic

73a030e

apache#17838 Extend predicate const evaluation

7ad94e1

pepijnve force-pushed the issue_17801 branch from 88dcd11 to 7ad94e1 Compare October 1, 2025 07:35

apache#17838 Correctly report nullability of implicit casts in predic…

7ff0810

…ates

		when(binary_expr(col("foo"), Operator::Eq, lit(5)), col("foo"))
		.otherwise(lit(0))?,

	when(binary_expr(col("foo"), Operator::Eq, lit(5)), col("foo"))
	.otherwise(lit(0))?,
	when(col("foo").eq(lit(5))), col("foo")).otherwise(lit(0))?,

-    fn check_nullability(
+    /// Verifies that `expr` has `nullable` nullability when the 'foo' column is
+    /// null.
+    /// Also assumes and verifies that `expr` is NOT nullable when 'foo' is NOT null
+    fn check_nullability(

#17801 Improve nullability reporting of case expressions #17813

Are you sure you want to change the base?

#17801 Improve nullability reporting of case expressions #17813

Conversation

pepijnve commented Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pepijnve commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pepijnve commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pepijnve commented Sep 30, 2025

Uh oh!

pepijnve commented Sep 30, 2025

Uh oh!

alamb commented Sep 30, 2025

Uh oh!

pepijnve commented Sep 30, 2025

Uh oh!

pepijnve commented Oct 1, 2025

Uh oh!

Uh oh!

pepijnve commented Sep 28, 2025 •

edited

Loading

pepijnve commented Sep 29, 2025 •

edited

Loading

pepijnve commented Sep 30, 2025 •

edited

Loading