[SPARK-57088] [SQL] Allow non-deterministic ranking expression for EXACT NEAREST BY

[zhidongqu-db]

What changes were proposed in this pull request?

Removes the NEAREST_BY_JOIN.EXACT_WITH_NONDETERMINISTIC_EXPRESSION rejection in CheckAnalysis so the EXACT mode of NEAREST BY JOIN (added in SPARK-56395) accepts non-deterministic ranking expressions, the same way APPROX already does.

Concretely:

• Drop the NearestByJoin arm in CheckAnalysis that failed analysis when approx = false and the ranking expression was non-deterministic.
• Change NearestByJoin.allowNonDeterministicExpression to return true unconditionally (was previously returning approx).
• Delete the NEAREST_BY_JOIN.EXACT_WITH_NONDETERMINISTIC_EXPRESSION error condition.
• Update scaladoc/comments in NearestByJoin and RewriteNearestByJoin to reflect that both modes permit a non-deterministic ranking expression.
• Update the user-facing docs in sql-ref-syntax-qry-select-join.md.
• Convert the existing rejection tests (Scala, Python, SQL golden) to positive tests asserting that EXACT + a non-deterministic ranking expression now succeeds.

Why are the changes needed?

APPROX vs. EXACT and determinism are orthogonal concerns:

• APPROX vs. EXACT is about the search algorithm contract: APPROX permits the optimizer to use faster approximate strategies (e.g. indexed ANN); EXACT forces brute-force evaluation.
• Determinism is a property of the ranking expression itself. Ordinary joins, for example, accept non-deterministic join conditions without forcing the user into an "approximate" join.

EXACT describes algebraic semantics ("compute the exact top-K according to the user's ranking expression"); it does not promise reproducibility across runs when the ranking expression is itself non-deterministic. Coupling the two was an over-restriction that this PR removes.

Does this PR introduce any user-facing change?

Yes. Queries of the form

SELECT ... FROM left JOIN right EXACT NEAREST k BY {DISTANCE | SIMILARITY} <non-deterministic expression>

previously failed at analysis with NEAREST_BY_JOIN.EXACT_WITH_NONDETERMINISTIC_EXPRESSION; they are now accepted and evaluated through the same brute-force rewrite as the APPROX variant.

The error condition NEAREST_BY_JOIN.EXACT_WITH_NONDETERMINISTIC_EXPRESSION is removed.

How was this patch tested?

• Updated DataFrameNearestByJoinSuite: the rejection test is converted to a positive test asserting the result count (21/21 passing locally).
• Updated the PySpark equivalent in test_nearest_by_join.py.
• Updated the SQL golden file join-nearest-by.sql (replaced the failing-EXACT query with a COUNT(*) query mirroring the existing APPROX case); regenerated results/ and analyzer-results/. SQLQueryTestSuite -z join-nearest-by passes (2/2).
• RewriteNearestByJoinSuite (12/12) still passes — the materializing-Project path in the optimizer rewrite already handled non-deterministic ranking expressions; only the analyzer gate changes.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.7)

[dtenedor]

LGTM, thanks for working on this.

[cloud-fan]

thanks, merging to master/4.x/4.2!