[SPARK-37716][SQL] Improve error messages when a LateralJoin has non-deterministic expressions

[allisonwang-db]

What changes were proposed in this pull request?

This PR allows the LateralJoin node's lateral subquery field to host non-deterministic expressions when the outer relation can produce at most one row. It also improves the error messages when a lateral join contains non-deterministic expressions that are not currently supported.

Why are the changes needed?

SPARK-37199 changes PlanExpression's deterministic field definition: both the children and the plan itself have to be deterministic for the plan expression to be deterministic. So users can no longer use lateral join with non-deterministic lateral subqueries. This PR is to improve the error messages and allows a special case when the outer query only produces at most one row.

Does this PR introduce any user-facing change?

Yes. Improve error messages:
Before:

org.apache.spark.sql.AnalysisException: nondeterministic expressions are only allowed in
Project, Filter, Aggregate or Window

After

Non-deterministic lateral subqueries are not supported when joining with outer relations that produce more than one row
-- Or
Lateral join condition cannot be non-deterministic:

How was this patch tested?

SQL query tests.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50968/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50970/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50968/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50970/

[SparkQA]

Test build #146492 has finished for PR 34987 at commit 58c3863.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[allisonwang-db]

cc @cloud-fan

[SparkQA]

Test build #146494 has finished for PR 34987 at commit 3c14c21.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51016/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51016/

[SparkQA]

Test build #146541 has finished for PR 34987 at commit 4822a61.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

This PR doesn't touch pyspark at all, and the github action failure is unrelated

starting mypy annotations test...
annotations failed mypy checks:

[cloud-fan]

merging to master, thanks!