Skip to content

[SPARK-56840][SQL] Avoid unresolved NullIf type lookup#55838

Closed
sunchao wants to merge 3 commits into
apache:masterfrom
sunchao:dev/chao/codex/oss-nullif-unresolved
Closed

[SPARK-56840][SQL] Avoid unresolved NullIf type lookup#55838
sunchao wants to merge 3 commits into
apache:masterfrom
sunchao:dev/chao/codex/oss-nullif-unresolved

Conversation

@sunchao
Copy link
Copy Markdown
Member

@sunchao sunchao commented May 12, 2026

Why are the changes needed?

NULLIF builds its replacement expression before analysis has resolved all child expressions.
For nested field references, the existing implementation can read the left operand's data type
too early while constructing the null branch, which can fail analysis even though the SQL shape
is valid.

SPARK-56840 tracks this analyzer failure.

What changes were proposed in this PR?

  • Build the NULLIF null branch with a lazy typed-null placeholder so construction does not eagerly
    read the unresolved left operand type, while NullIf.replacement.dataType remains valid once the
    operand type is available.
  • Make that placeholder RuntimeReplaceable, so ReplaceExpressions restores an ordinary typed
    Literal(null, ...) before later optimizer rules run and existing null-literal simplifications
    continue to apply.
  • Add focused regressions for:
    • nested struct-field nullif(c.provider, lower(...)) analysis in both
      ALWAYS_INLINE_COMMON_EXPR modes;
    • NullIf replacement type reporting before type coercion;
    • optimizer replacement back to a normal null literal;
    • explain output avoiding exposure of the internal helper name.

Does this PR introduce any user-facing change?

Yes. Valid NULLIF expressions over unresolved nested field references that could fail during
analysis now resolve and execute successfully.

How was this patch tested?

  • build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.NullExpressionsSuite -- -z "NullIf replacement preserves its data type before type coercion"'
  • build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizerSuite -- -z "NullIf typed null branch is replaced with a null literal"'
  • build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z "nullif function"'
  • build/sbt 'sql/testOnly org.apache.spark.sql.ExplainSuite -- -z "explain for these functions; use range to avoid constant folding"'

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5.5)

Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @sunchao .

cc @peter-toth .

peter-toth pushed a commit that referenced this pull request May 14, 2026
### Why are the changes needed?

`NULLIF` builds its replacement expression before analysis has resolved all child expressions.
For nested field references, the existing implementation can read the left operand's data type
too early while constructing the null branch, which can fail analysis even though the SQL shape
is valid.

SPARK-56840 tracks this analyzer failure.

### What changes were proposed in this PR?

- Build the `NULLIF` null branch with a lazy typed-null placeholder so construction does not eagerly
  read the unresolved left operand type, while `NullIf.replacement.dataType` remains valid once the
  operand type is available.
- Make that placeholder `RuntimeReplaceable`, so `ReplaceExpressions` restores an ordinary typed
  `Literal(null, ...)` before later optimizer rules run and existing null-literal simplifications
  continue to apply.
- Add focused regressions for:
  - nested struct-field `nullif(c.provider, lower(...))` analysis in both
    `ALWAYS_INLINE_COMMON_EXPR` modes;
  - `NullIf` replacement type reporting before type coercion;
  - optimizer replacement back to a normal null literal;
  - explain output avoiding exposure of the internal helper name.

### Does this PR introduce _any_ user-facing change?

Yes. Valid `NULLIF` expressions over unresolved nested field references that could fail during
analysis now resolve and execute successfully.

### How was this patch tested?

- `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.NullExpressionsSuite -- -z "NullIf replacement preserves its data type before type coercion"'`
- `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizerSuite -- -z "NullIf typed null branch is replaced with a null literal"'`
- `build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z "nullif function"'`
- `build/sbt 'sql/testOnly org.apache.spark.sql.ExplainSuite -- -z "explain for these functions; use range to avoid constant folding"'`

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5.5)

Closes #55838 from sunchao/dev/chao/codex/oss-nullif-unresolved.

Authored-by: Chao Sun <sunchao@apache.org>
Signed-off-by: Peter Toth <peter.toth@gmail.com>
(cherry picked from commit 5949ab3)
Signed-off-by: Peter Toth <peter.toth@gmail.com>
peter-toth pushed a commit that referenced this pull request May 14, 2026
### Why are the changes needed?

`NULLIF` builds its replacement expression before analysis has resolved all child expressions.
For nested field references, the existing implementation can read the left operand's data type
too early while constructing the null branch, which can fail analysis even though the SQL shape
is valid.

SPARK-56840 tracks this analyzer failure.

### What changes were proposed in this PR?

- Build the `NULLIF` null branch with a lazy typed-null placeholder so construction does not eagerly
  read the unresolved left operand type, while `NullIf.replacement.dataType` remains valid once the
  operand type is available.
- Make that placeholder `RuntimeReplaceable`, so `ReplaceExpressions` restores an ordinary typed
  `Literal(null, ...)` before later optimizer rules run and existing null-literal simplifications
  continue to apply.
- Add focused regressions for:
  - nested struct-field `nullif(c.provider, lower(...))` analysis in both
    `ALWAYS_INLINE_COMMON_EXPR` modes;
  - `NullIf` replacement type reporting before type coercion;
  - optimizer replacement back to a normal null literal;
  - explain output avoiding exposure of the internal helper name.

### Does this PR introduce _any_ user-facing change?

Yes. Valid `NULLIF` expressions over unresolved nested field references that could fail during
analysis now resolve and execute successfully.

### How was this patch tested?

- `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.NullExpressionsSuite -- -z "NullIf replacement preserves its data type before type coercion"'`
- `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizerSuite -- -z "NullIf typed null branch is replaced with a null literal"'`
- `build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z "nullif function"'`
- `build/sbt 'sql/testOnly org.apache.spark.sql.ExplainSuite -- -z "explain for these functions; use range to avoid constant folding"'`

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5.5)

Closes #55838 from sunchao/dev/chao/codex/oss-nullif-unresolved.

Authored-by: Chao Sun <sunchao@apache.org>
Signed-off-by: Peter Toth <peter.toth@gmail.com>
(cherry picked from commit 5949ab3)
Signed-off-by: Peter Toth <peter.toth@gmail.com>
peter-toth pushed a commit that referenced this pull request May 14, 2026
### Why are the changes needed?

`NULLIF` builds its replacement expression before analysis has resolved all child expressions.
For nested field references, the existing implementation can read the left operand's data type
too early while constructing the null branch, which can fail analysis even though the SQL shape
is valid.

SPARK-56840 tracks this analyzer failure.

### What changes were proposed in this PR?

- Build the `NULLIF` null branch with a lazy typed-null placeholder so construction does not eagerly
  read the unresolved left operand type, while `NullIf.replacement.dataType` remains valid once the
  operand type is available.
- Make that placeholder `RuntimeReplaceable`, so `ReplaceExpressions` restores an ordinary typed
  `Literal(null, ...)` before later optimizer rules run and existing null-literal simplifications
  continue to apply.
- Add focused regressions for:
  - nested struct-field `nullif(c.provider, lower(...))` analysis in both
    `ALWAYS_INLINE_COMMON_EXPR` modes;
  - `NullIf` replacement type reporting before type coercion;
  - optimizer replacement back to a normal null literal;
  - explain output avoiding exposure of the internal helper name.

### Does this PR introduce _any_ user-facing change?

Yes. Valid `NULLIF` expressions over unresolved nested field references that could fail during
analysis now resolve and execute successfully.

### How was this patch tested?

- `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.NullExpressionsSuite -- -z "NullIf replacement preserves its data type before type coercion"'`
- `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizerSuite -- -z "NullIf typed null branch is replaced with a null literal"'`
- `build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z "nullif function"'`
- `build/sbt 'sql/testOnly org.apache.spark.sql.ExplainSuite -- -z "explain for these functions; use range to avoid constant folding"'`

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5.5)

Closes #55838 from sunchao/dev/chao/codex/oss-nullif-unresolved.

Authored-by: Chao Sun <sunchao@apache.org>
Signed-off-by: Peter Toth <peter.toth@gmail.com>
(cherry picked from commit 5949ab3)
Signed-off-by: Peter Toth <peter.toth@gmail.com>
peter-toth pushed a commit that referenced this pull request May 14, 2026
### Why are the changes needed?

`NULLIF` builds its replacement expression before analysis has resolved all child expressions.
For nested field references, the existing implementation can read the left operand's data type
too early while constructing the null branch, which can fail analysis even though the SQL shape
is valid.

SPARK-56840 tracks this analyzer failure.

### What changes were proposed in this PR?

- Build the `NULLIF` null branch with a lazy typed-null placeholder so construction does not eagerly
  read the unresolved left operand type, while `NullIf.replacement.dataType` remains valid once the
  operand type is available.
- Make that placeholder `RuntimeReplaceable`, so `ReplaceExpressions` restores an ordinary typed
  `Literal(null, ...)` before later optimizer rules run and existing null-literal simplifications
  continue to apply.
- Add focused regressions for:
  - nested struct-field `nullif(c.provider, lower(...))` analysis in both
    `ALWAYS_INLINE_COMMON_EXPR` modes;
  - `NullIf` replacement type reporting before type coercion;
  - optimizer replacement back to a normal null literal;
  - explain output avoiding exposure of the internal helper name.

### Does this PR introduce _any_ user-facing change?

Yes. Valid `NULLIF` expressions over unresolved nested field references that could fail during
analysis now resolve and execute successfully.

### How was this patch tested?

- `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.NullExpressionsSuite -- -z "NullIf replacement preserves its data type before type coercion"'`
- `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizerSuite -- -z "NullIf typed null branch is replaced with a null literal"'`
- `build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z "nullif function"'`
- `build/sbt 'sql/testOnly org.apache.spark.sql.ExplainSuite -- -z "explain for these functions; use range to avoid constant folding"'`

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5.5)

Closes #55838 from sunchao/dev/chao/codex/oss-nullif-unresolved.

Authored-by: Chao Sun <sunchao@apache.org>
Signed-off-by: Peter Toth <peter.toth@gmail.com>
(cherry picked from commit 5949ab3)
Signed-off-by: Peter Toth <peter.toth@gmail.com>
@peter-toth
Copy link
Copy Markdown
Contributor

peter-toth commented May 14, 2026

@sunchao , sorry, I think I merged this PR a bit too early, can you please doublecheck the repro test as it seems to pass without the fix on master?

I think the issue is real, but we probably need a proper repro in a follow-up PR.

Also, I couldn't merge this to branch-3.5 due to conflicts, can you please open a backport PR as well?

@sunchao
Copy link
Copy Markdown
Member Author

sunchao commented May 14, 2026

Thanks @peter-toth . Let me take a look on the repro test. And yes, I can open a PR against branch-3.5 also.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants