[SPARK-32801][SQL] Make InferFiltersFromConstraints take into account EqualNullSafe

[tanelk]

What changes were proposed in this pull request?

InferFiltersFromConstraints only infers new filters using EqualTo, generalized it to also include EqualNullSafe.

This introduced an infinite loop in InferFiltersFromConstraintsSuite. It has been fixed in the Optimizer by restructuring the batches (#19149) - I did the same in the test suite.

Also it revealed, that the InferFiltersFromConstraints is not idempotent, for two cases:

• A chain of filters like "x.a".attr === "x.b".attr && "x.a".attr === "y.a".attr && "x.b".attr === "y.b".attr, that should infer "y.a".attr === "y.b".attr
• Constraints form the right side of left semi join are ignored when it is a child of a another join

These were also fixed

Why are the changes needed?

Possible performance improvement from new inferred constraints

Does this PR introduce any user-facing change?

No

How was this patch tested?

UT

[wangyum]

Are you sure we can infer from EqualNullSafe? Some EqualNullSafe from Alias:

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala

Lines 175 to 183 in 0a6043f

	case a @ Alias(l: Literal, _) =>
	allConstraints += EqualNullSafe(a.toAttribute, l)
	case a @ Alias(e, _) =>
	// For every alias in `projectList`, replace the reference in constraints by its attribute.
	allConstraints ++= allConstraints.map(_ transform {
	case expr: Expression if expr.semanticEquals(e) =>
	a.toAttribute
	})
	allConstraints += EqualNullSafe(e, a.toAttribute)

[tanelk]

Are you sure we can infer from EqualNullSafe? Some EqualNullSafe from Alias:

I do not follow, why should this make EqualNullSafe not work?

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37659/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37659/

[tanelk]

@cloud-fan, it seems you worked on this
also cc @maropu

[SparkQA]

Test build #133059 has finished for PR 29650 at commit 6d56655.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #134102 has finished for PR 29650 at commit 6d56655.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

cc @maryannxue too

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!