fix(spark): match Spark abs overflow errors - #24156
Open
goutamadwant wants to merge 2 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
When ANSI mode is enabled,
datafusion-sparkreports type-specific DataFusion errors for integralabsoverflow. The scalar and array paths also produce different messages. Spark 4.2 uses one canonicalARITHMETIC_OVERFLOWmessage for both paths.What changes are included in this PR?
absarray kernel so Spark-specific error text does not change core DataFusion behavior.ARITHMETIC_OVERFLOWmessage for scalar and array overflows.Are these changes tested?
Yes. The following checks pass:
cargo fmt --all -- --checkcargo clippy -p datafusion-spark --all-targets --all-features -- -D warningscargo test -p datafusion-spark --all-featurescargo test --profile=ci --test sqllogictests -- spark/math/abs.sltcargo test --profile=ci --test sqllogictests -- math.sltRUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryptionAre there any user-facing changes?
Yes. With
datafusion.execution.enable_ansi_mode = true, Sparkabsoverflow errors now use Spark 4.2's canonicalARITHMETIC_OVERFLOWmessage for scalar and array inputs. There are no public API changes.