fix/substrait-lambda-rextype-0.62.3+

[KARTIK64-rgb]

Which issue does this PR close?

• Closes Latest substrait patch release does not compile against DF #20756.

Rationale for this change

The substrait crate released version 0.62.3 which introduced two new RexType enum variants, Lambda and LambdaInvocation, as a breaking change from 0.62.2. Since DataFusion's workspace Cargo.toml specifies the substrait dependency as "0.62", Cargo resolves to 0.62.3 on a clean checkout, causing datafusion-substrait to fail to compile entirely with the following error:

error[E0004]: non-exhaustive patterns: `&RexType::Lambda(_)` and `&RexType::LambdaInvocation(_)` not covered

This means anyone building datafusion-substrait from a clean environment today will hit a hard compile failure with no workaround.

What changes are included in this PR?

Two new match arms have been added to the from_substrait_rex function in datafusion/substrait/src/logical_plan/consumer/expr/mod.rs to handle the two new variants introduced in substrait 0.62.3:

            RexType::Lambda(_) => {
                not_impl_err!("Lambda expressions are not yet supported")
            }
            RexType::LambdaInvocation(_) => {
                not_impl_err!("LambdaInvocation expressions are not yet supported")
            }
        },

Both arms return not_impl_err! with a descriptive message, which is exactly the same pattern already used in this codebase for other unsupported expression types such as Nested, Enum, MultiOrList, and DynamicParameter. No existing behaviour was changed and no other files were modified.

Are these changes tested?

The fix restores compilation against substrait 0.62.3. No new unit tests are added because Lambda and LambdaInvocation are not yet supported at a functional level. The not_impl_err! return ensures that any Substrait plan containing these variants will produce a clean and descriptive DataFusion error: This feature is not implemented message at runtime, consistent with how all other unsupported expression types in this codebase behave. The full existing test suite was run locally with cargo test -p datafusion-substrait and all existing tests continue to pass.

Are there any user-facing changes?

Prior to this fix, datafusion-substrait would not compile at all when Cargo resolved substrait to version 0.62.3. After this fix, the crate compiles correctly against 0.62.3. Any Substrait plan containing a Lambda or LambdaInvocation expression will return a clear not-implemented error at runtime rather than a compile failure. There are no breaking API changes.

[milenkovicm]

perhaps we should pin version to exact as well? not sure what @gabotechs thinks

[milenkovicm]

I would suggest to pin version to 0.62.2 for this release and then increase it later, wdyt @AdamGS ?

[milenkovicm]

this is duplicate of #20750

[AdamGS]

whichever PR is merged, I think the version should be pinned ("-0.62.3") as it seems substrait's versioning isn't semver compatible? it might be possible to change that enum to #[non_exahustive] upstream

[benbellick]

This particular release of substrait-rs may have been botched by me 😅. I am going to investigate and get back to all of you shortly.

[milenkovicm]

#19692 (comment) it might affect 53 release

[benbellick]

This release of substrait-rs breaks semvar compatibility. This was an oversight on my end due to the fact that release-plz didn't detect a submodule bump as a breaking change.

We are investigating performing cargo yank on this release. Would that solution work for others here?

[milenkovicm]

I have pinned version of substrait for df.53 in #20827 if no better suggestion

[milenkovicm]

This release of substrait-rs breaks semvar compatibility. This was an oversight on my end due to the fact that release-plz didn't detect a submodule bump as a breaking change.

We are investigating performing cargo yank on this release. Would that solution work for others here?

This is going to be paint point with 52.3, do you plan to yank it @benbellick ?

[benbellick]

@milenkovicm yes that is the plan right now. Unfortunately, I do not have credentials to yank that package. I have contracted the PMCs that have access and hopefully should be able to get this resolved quickly.

[gabotechs]

I would suggest to pin version to 0.62.2

The challenge with pinning the version to a specific patch version is that it might force an unintended dependency change to the people depending on both datafusion an substrait.

I see that in #20827 (comment) we have not really pinned the version to 0.62.2, the change there is still satisfied by 0.62.3.

My impression is that the proper solution is to yank substrait-rs 0.62.3 release, remove it from crates.io, do a major release, as it's a breaking API change.