[SPARK-57005][SQL] Fix None.get in RewritePredicateSubquery when subquery predicates are eliminated

[yadavay-amzn]

What changes were proposed in this pull request?

Guard joinCond.get with isDefined check in rewriteDomainJoinsIfPresent.

Why are the changes needed?

When constant folding eliminates all correlated predicates in a subquery (e.g., WHERE FALSE AND correlated_pred), rewriteExistentialExpr returns None for joinCond. The subsequent call to rewriteDomainJoinsIfPresent then crashes with NoSuchElementException: None.get.

Repro:

SELECT t0.value FROM VALUES (CAST(0 AS TINYINT)) AS t0(value)
WHERE NOT EXISTS (
  SELECT t1.value FROM VALUES (CAST(0 AS TINYINT)) AS t1(value)
  WHERE FALSE AND (t0.value = t1.value)
)

Does this PR introduce any user-facing change?

Yes -- queries with EXISTS/NOT EXISTS subqueries where all correlated predicates are eliminated by constant folding no longer crash.

How was this patch tested?

Added test in SubquerySuite covering both NOT EXISTS (returns all rows) and EXISTS (returns no rows) with WHERE FALSE AND (correlated_pred). Test verifies correct query semantics, not just absence of crash.

• Without fix: NoSuchElementException: None.get at subquery.scala:76
• With fix: correct results returned
• Full SubquerySuite (142 tests) passes with no regressions.

Was this patch authored or co-authored using generative AI tooling?

Generated by: Claude Opus 4.7

[eejbyfeldt]

Would it be possible to add the test in RewriteSubquerySuite instead? So that it more localized and faster to run since it does not have to bring in the full planning machinery.

[yadavay-amzn]

Done -- moved the test to RewriteSubquerySuite as a catalyst-level test using the optimizer rules directly. Thanks @eejbyfeldt!

[eejbyfeldt]

The pullup rule is not needed here to reproduce the crash? What the motivation for not just using the simpler Optimize already defined?

Suggested change

	// When BooleanSimplification in PullupCorrelatedPredicates eliminates all correlated
	// predicates (e.g., FALSE AND correlated_pred -> FALSE), the Exists node ends up with
	// outerAttrs non-empty but joinCond empty. RewritePredicateSubquery must handle this.
	object OptimizeWithPullup extends RuleExecutor[LogicalPlan] {
	val batches =
	Batch("Pullup Correlated Expressions", Once,
	PullupCorrelatedPredicates) ::
	Batch("Rewrite Subquery", FixedPoint(1),
	RewritePredicateSubquery,
	ColumnPruning,
	CollapseProject,
	RemoveNoopOperators) :: Nil
	}
	val outer = LocalRelation($"a".int, $"b".int)
	val inner = LocalRelation($"x".int, $"y".int)
	// NOT EXISTS with FALSE AND correlated_pred - should not throw None.get
	val notExistsQuery = outer.where(
	Not(Exists(inner.where(Literal.FalseLiteral && $"a" === $"x")))).select($"a")
	OptimizeWithPullup.execute(notExistsQuery.analyze)
	// EXISTS with FALSE AND correlated_pred - should not throw None.get
	val existsQuery = outer.where(
	Exists(inner.where(Literal.FalseLiteral && $"a" === $"x"))).select($"a")
	OptimizeWithPullup.execute(existsQuery.analyze)
	val outer = LocalRelation($"a".int, $"b".int)
	val inner = LocalRelation($"x".int, $"y".int)
	// NOT EXISTS with FALSE AND correlated_pred - should not throw None.get
	val notExistsQuery = outer.where(
	Not(Exists(inner.where(Literal.FalseLiteral && $"a" === $"x")))).select($"a")
	Optimize.execute(notExistsQuery.analyze)
	// EXISTS with FALSE AND correlated_pred - should not throw None.get
	val existsQuery = outer.where(
	Exists(inner.where(Literal.FalseLiteral && $"a" === $"x"))).select($"a")
	Optimize.execute(existsQuery.analyze)