[SPARK-17075][SQL][followup] Add Estimation of Constant Literal

[gatorsmile]

What changes were proposed in this pull request?

FalseLiteral and TrueLiteral should have been eliminated by optimizer rule BooleanSimplification, but null literals might be added by optimizer rule NullPropagation. For safety, our filter estimation should handle all the eligible literal cases.

Our optimizer rule BooleanSimplification is unable to remove the null literal in many cases. For example, a < 0 or null. Thus, we need to handle null literal in filter estimation.

Not can be pushed down below And and Or. Then, we could see two consecutive Not, which need to be collapsed into one. Because of the limited expression support for filter estimation, we just need to handle the case Not(null) for avoiding incorrect error due to the boolean operation on null. For details, see below matrix.

not NULL = NULL
NULL or false = NULL
NULL or true = true
NULL or NULL = NULL
NULL and false = false
NULL and true = NULL
NULL and NULL = NULL

How was this patch tested?

Added the test cases.

[gatorsmile]

cc @ron8hu @cloud-fan @wzhfy

[gatorsmile]

Ideally, our optimizer rule BooleanSimplification should be able to remove the null literal, like what we did for true and false literals. Here, I think we should just assume our filter estimation does not depend on any other optimizer rules.

[SparkQA]

Test build #75289 has finished for PR 17446 at commit d770dbb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #75291 has finished for PR 17446 at commit 1775ca7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[ron8hu]

The logic is straightforward. LGTM.

[gatorsmile]

Thank you! @ron8hu

[cloud-fan]

handling null in filter estimation is not trivial, e.g. null and false returns false, null and true returns true. If we estimate cond && null, we will report 0 selectivity, which is wrong.

I think we should eliminate null literal in optimizer when it's involved in filter condition.

[gatorsmile]

Yes, let me close it.

not NULL = NULL
NULL or false = NULL
NULL or true = true
NULL or NULL = NULL
NULL and false = false
NULL and true = NULL
NULL and NULL = NULL

[gatorsmile]

Wait... It behaves correctly, right?

[wzhfy]

Ideally, our optimizer rule BooleanSimplification should be able to remove the null literal, like what we did for true and false literals. Here, I think we should just assume our filter estimation does not depend on any other optimizer rules.

Sorry I may miss some context, but I prefer enhancing BooleanSimplification to deal with null literals. There's no need to complicate estimation logic for those deterministic literals which can be removed by optimizer.

[gatorsmile]

@wzhfy BooleanSimplification is unable to get rid of null in many cases. Boolean operations on null is complex. We need more investigation.

[gatorsmile]

Since this PR does not correctly handle the cases like Not(null), I close this PR at first.

[gatorsmile]

null should not be simply treated as a false literal in filter estimation. Based on the definition, Not(null) should return null. If we treat null as false, Not(null) will return 1.0, which is wrong in many cases.

[wzhfy]

LGTM

[SparkQA]

Test build #75348 has finished for PR 17446 at commit 8ec57f3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

Thanks! Merging to master.