Skip to content

fix: Emit equality conditions for Substrait CASE base expressions - #25191

Open
namanjain24-sudo wants to merge 1 commit into
apache:mainfrom
namanjain24-sudo:fix-substrait-case-base-expression
Open

fix: Emit equality conditions for Substrait CASE base expressions#25191
namanjain24-sudo wants to merge 1 commit into
apache:mainfrom
namanjain24-sudo:fix-substrait-case-base-expression

Conversation

@namanjain24-sudo

@namanjain24-sudo namanjain24-sudo commented Sep 11, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

Substrait's IfThen has no base expression. The spec writes every clause as if <boolean expression> then <result expression>.

The producer used IfThen for CASE <base> WHEN <value> anyway, through a convention private to DataFusion: a first clause holding the base in if with then left unset, then one clause per WHEN carrying the raw WHEN operand. SELECT CASE a WHEN 1 THEN 'x' WHEN 2 THEN 'y' ELSE 'z' END came out as three clauses whose conditions are the field reference a and the literals 1 and 2. None is boolean, and the first has no result. Our consumer reads the convention back, so round trips were unaffected and no test failed. Another engine sees clauses it cannot evaluate.

What changes are included in this PR?

from_case now emits one clause per WHEN, with <base> = <value> as the condition. That is the desugaring from_between already uses for BETWEEN, and it is how the spec describes a switch: an if expression whose conditions are all equality against the same value. The searched form is unchanged. Meaning is preserved, because DataFusion matches a base CASE with the same equality kernels = lowers to.

Left out deliberately:

  • SwitchExpression itself. Its IfValue.if is a Literal, so it cannot hold CASE a WHEN b + 1, and our consumer answers not_impl_err!("Switch expression not supported").
  • The consumer, which still accepts the old encoding, so plans written by older DataFusion versions keep reading.

The cost is that the base expression is repeated once per WHEN arm, and evaluated per arm.

What is the testing strategy for this PR?

case_with_base_expression_emits_equality_conditions inspects the produced protobuf instead of round-tripping, because the consumer understands the old encoding and a round trip cannot catch this. case_with_base_expression moves to assert_expected_plan to record the new shape, still asserting the schema is unchanged.

Are there any user-facing changes?

A base CASE now serialises to equal calls with a boolean output type. No Rust API changes. Inside DataFusion the plan round trips into the equivalent searched CASE, with the same schema and results. A consumer that implemented the old convention sees the new form, which is valid Substrait.

@github-actions github-actions Bot added the substrait Changes to the substrait crate label Sep 11, 2026
@namanjain24-sudo

Copy link
Copy Markdown
Author

@gabotechs @kosiew when you have a moment, would one of you be able to trigger the workflows on this PR? The contributor guide notes a committer has to do that for a new contributor, and no checks have run here yet.

Locally this passes cargo test -p datafusion-substrait (272 tests), cargo xtask ci step test substrait, and the rust_clippy.sh, rust_fmt.sh, typos_check.sh and rust_docs.sh steps of the lint suite.

On the change itself: the one judgement call is desugaring CASE <base> WHEN <value> into <base> = <value> conditions rather than emitting SwitchExpression. I went that way because SwitchExpression.IfValue.if is a Literal, so it cannot express a non-literal WHEN operand, and our consumer currently answers not_impl_err!("Switch expression not supported"), so emitting it would break our own round trip. The trade-off is that the base expression is now repeated once per WHEN arm. I described both in #25190 and would be glad to take the PR in a different direction if you'd prefer.

@namanjain24-sudo
namanjain24-sudo force-pushed the fix-substrait-case-base-expression branch from b9ff008 to 2848da2 Compare September 11, 2026 16:02
Substrait's `IfThen` has no base expression. Every `IfClause` is a
standalone boolean condition, and `then` is the value that clause yields.

The producer instead encoded `CASE <base> WHEN <value> THEN ...` by pushing
a leading `IfClause` that carries the base expression in `if` and leaves
`then` unset, followed by one clause per WHEN whose `if` is the raw WHEN
operand. For `CASE a WHEN 1 THEN 'x' WHEN 2 THEN 'y' ELSE 'z' END` that
emits three clauses whose conditions are `a`, `1` and `2`, none of which is
boolean, and a first clause with no result.

The convention is private to DataFusion: the consumer reads a `then`-less
first clause back as the base expression, so a DataFusion-to-DataFusion
round trip is unaffected. Any other engine sees clauses it cannot evaluate.

Emit `<base> = <value>` as each clause condition instead, the same
desugaring `from_between` already applies to `BETWEEN`. DataFusion matches
a base expression with `=` semantics, so the plan keeps its meaning,
including a NULL WHEN operand never matching.

A base `CASE` now round trips as the equivalent searched `CASE`, keeping
its original projection name and schema.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.93%. Comparing base (f8cc678) to head (b87b857).

Files with missing lines Patch % Lines
...ubstrait/src/logical_plan/producer/expr/if_then.rs 75.00% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25191      +/-   ##
==========================================
- Coverage   81.93%   81.93%   -0.01%     
==========================================
  Files        1133     1133              
  Lines      423529   423529              
  Branches   423529   423529              
==========================================
- Hits       347028   347018      -10     
- Misses      55910    55920      +10     
  Partials    20591    20591              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gabotechs

Copy link
Copy Markdown
Contributor

Hi @namanjain24-sudo, thanks for the PR!

Could I ask to summarize and condense a bit the PR description? Feel free to keep anything that is not already implicit in the code itself, aim for a 2 or 3 min read summary. That would help reviewers better digest the PR. Thanks!

@namanjain24-sudo

Copy link
Copy Markdown
Author

Thanks, trimmed it. What I cut (the local test log, and the longer reasoning for leaving SwitchExpression alone) is written up in #25190 if it comes up in review.

On the red check, I don't think it's this PR: it's cargo test datafusion-cli, which has no dependency on datafusion-substrait. MinIO withdrew minio/minio from Docker Hub on the 11th, so the storage integration tests panic on startup, and my branch is based on a commit from before #25216 moved that pull to quay.io. Rebasing onto main should clear it, happy to do that whenever, though it will probably need another workflow approval from you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Substrait producer emits non-boolean IfThen clauses for CASE with a base expression

3 participants